pereinar 2002/06/05 10:39:53
Modified: src/docs/tutorials Changes.pod config.cfg
Added: src/docs/tutorials/apps/scale_etoys app_servers.png
code_structure.png etoys.pod machine_layout.png
proxy_architecture.png proxy_servers.png
search_servers.png session_tracking.png
src/docs/tutorials/client/browserbugs browserbugs.pod
src/docs/tutorials/tips/mod_perl_tricks mod_perl_tricks.pod
src/docs/tutorials/tmpl/comparison comparison.pod
Removed: src/docs/tutorials/browserbugs browserbugs.pod
src/docs/tutorials/mod_perl_tricks mod_perl_tricks.pod
src/docs/tutorials/scale_etoys app_servers.png
code_structure.png etoys.pod machine_layout.png
proxy_architecture.png proxy_servers.png
search_servers.png session_tracking.png
src/docs/tutorials/templates comparison.pod
Log:
Re-organized tutorials section to prepare for potential new tutorials.
Revision Changes Path
1.4 +4 -1 modperl-docs/src/docs/tutorials/Changes.pod
Index: Changes.pod
===================================================================
RCS file: /home/cvs/modperl-docs/src/docs/tutorials/Changes.pod,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -r1.3 -r1.4
--- Changes.pod 3 Jun 2002 16:09:45 -0000 1.3
+++ Changes.pod 5 Jun 2002 17:39:52 -0000 1.4
@@ -9,7 +9,10 @@
The most recent changes are listed first.
-=head1 ...
+=head1 Wed Jun 5 19:24:05 CET 2002
+
+* Re-arranged content completely to better classify and be able to
+ adjust to more tutorials. [Per Einar]
* browserbugs::browserbugs: Added discussion of length of error
responses and how it's handled by IE, in relation to this
1.6 +23 -5 modperl-docs/src/docs/tutorials/config.cfg
Index: config.cfg
===================================================================
RCS file: /home/cvs/modperl-docs/src/docs/tutorials/config.cfg,v
retrieving revision 1.5
retrieving revision 1.6
diff -u -r1.5 -r1.6
--- config.cfg 29 May 2002 08:06:43 -0000 1.5
+++ config.cfg 5 Jun 2002 17:39:52 -0000 1.6
@@ -11,16 +11,34 @@
mod_perl programmers.
EOB
+ group => 'Application Design',
+ chapters => [qw(
+
+ apps/scale_etoys/etoys.pod
+ )],
+
+ group => 'Templating',
+ chapters => [qw(
+ tmpl/comparison/comparison.pod
+ )],
+
+ group => 'Tips and Tricks',
+ chapters => [qw(
+ tips/mod_perl_tricks/mod_perl_tricks.pod
+ )],
+
+ group => 'Client side facts and bugs',
+ chapters => [qw(
+ client/browserbugs/browserbugs.pod
+ )],
+
+ group => 'Miscellaneous',
chapters => [qw(
- templates/comparison.pod
- mod_perl_tricks/mod_perl_tricks.pod
- scale_etoys/etoys.pod
- browserbugs/browserbugs.pod
Changes.pod
)],
copy_glob => [qw(
- */*.png
+ */*/*.png
)],
);
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/app_servers.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/code_structure.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/etoys.pod
Index: etoys.pod
===================================================================
=head1 NAME
Building a Large-Scale E-commerce site with Apache and mod_perl
=head1 Description
mod_perl's speed and Perl's flexibility make them very attractive for
large-scale sites. Through careful planning from the start, powerful
application servers can be created for sites requiring excellent
response times for dynamic content, such as EToys, all by using
mod_perl.
This paper was first presented at ApacheCon 2001 in Santa Clara,
California, and was later published by O'Reilly & Associates' Perl.com
site: http://perl.com/pub/a/2001/10/17/etoys.html
=head1 Common Myths
When it comes to building a large e-commerce web site, everyone is
full of advice. Developers will tell you that only a site built in C++
or Java (depending on which they prefer) can scale up to handle heavy
traffic. Application server vendors will insist that you need a
packaged all-in-one solution for the software. Hardware vendors will
tell you that you need the top-of-the-line mega-machines to run a
large site. This is a story about how we built a large e-commerce site
using mainly open source software and commodity hardware. We did it,
and you can do it too.
=head1 Perl Saves
Perl has long been the preferred language for developing CGI scripts.
It combines supreme flexibility with rapid development. I<Programming
Perl> is still one of O'Reilly's top selling technical books, and
community support abounds. Lately though, Perl has come under attack
from certain quarters. Detractors claim that it's too slow for serious
development work and that code written in Perl is too hard to
maintain.
The mod_perl Apache module changes the whole performance picture for
Perl. Embedding a Perl interpreter inside of Apache provides
performance equivalent to Java servlets, and makes it an excellent
choice for building large sites. Through the use of Perl's
object-oriented features and some basic coding rules, you can build a
set of code that is a pleasure to maintain, or at least no worse than
other languages.
=head2 Roll Your Own Application Server
When you combine Apache, mod_perl, and open source code available from
CPAN (the Comprehensive Perl Archive Network), you get a set of
features equivalent to a commercial application server:
=over
=item *
Session handling
=item *
Load balancing
=item *
Persistent database connections
=item *
Advanced HTML templating
=item *
Security
=back
You also get some things you won't get from a commercial product, like
a direct line to the core development team through the appropriate
mailing list, and the ability to fix problems yourself instead of
waiting for a patch. Moreover, every part of the system is under your
control, making you limited only by your team's abilities.
=head1 Case Study: eToys.com
When we first arrived at eToys in 1999, we found a situation that is
probably familiar to many who have joined a growing startup Internet
company. The system was based on CGI scripts talking to a MySQL
database. Static file serving and dynamic content generation were
sharing resources on the same machines. The CGI code was largely
written in a Perl4-ish style and not as modular as it could be, which
was not surprising since most of it was built as quickly as possible
by a very small team.
Our major task was to figure out how to get this system to scale large
enough to handle the expected Christmas traffic. The toy business is
all about seasonality, and the difference between the peak selling
season and the rest of the year is enormous. The site had barely
survived the previous Christmas, and the MySQL database didn't look
like it could scale much further.
The call had already been made to switch to Oracle, and a DBA team was
in place. We didn't have enough time to do a re-design of the
software, so we had to scramble to put in place whatever performance
improvements we could finish by Christmas.
=head2 Apache::PerlRun to the Rescue
C<Apache::PerlRun> is a module that exists to smooth the transition
between basic CGI and mod_perl. It emulates a CGI environment, and
provides some (but not all) of the performance benefits associated
with code written for mod_perl. Using this module and the persistent
database connections provided by C<Apache::DBI>, we were able to do a
basic port to mod_perl and Oracle in time for Christmas, and combined
with some new hardware we were ready to face the Christmas rush.
The peak traffic lasted for eight weeks, most of which were spent
frantically fixing things or nervously waiting for something else to
break. Nevertheless, we made it through. During that time we collected
the following statistics:
=over
=item *
60 - 70,000 sessions/hour
=item *
800,000 page views/hour
=item *
7,000 orders/hour
=back
According to Media Metrix, we were the third most heavily trafficked
e-commerce site, right behind eBay and Amazon.
=head2 Planning the New Architecture
It was clear that we would need to do a re-design for 2000. We had
reached the limits of the current system and needed to tackle some of
the harder problems that we had been holding off on.
Goals for the new system included moving away from off-line page
generation. The old system had been building HTML pages for every
product and product category on the site in a batch job and dumping
them out as static files. This was very effective when we had a small
database of products since the static files gave such good
performance, but we had recently added a children's bookstore to the
site, which increased the size of our product database by an order of
magnitude and made the time required to generate every page
prohibitive. We needed a strategy that would only require us to build
pages that customers were actually interested in and would still
provide solid performance.
We also wanted to re-do the database schema for more flexibility, and
structure the code in a more modular way that would make it easier for
a team to share the development work without stepping on each other. We
knew that the new codebase would have to be flexible enough to support
a continuously evolving set of features.
Not all of the team had significant experience with object-oriented
Perl, so we brought in Randal Schwartz and Damian Conway to do
training sessions with us. We created a set of coding standards,
drafted a design, and built our system.
=head1 Surviving Christmas 2000
Our capacity planning was for three times the traffic of the previous
peak. That's what we tested to, and that's about what we got:
=over
=item *
200,000+ sessions/hour
=item *
2.5 million+ page views/hour
=item *
20,000+ orders/hour
=back
The software survived, although one of the routers went up in smoke.
Once again, we were rated the third most highly trafficked e-commerce
site for the season.
=head2 The Architecture
The machine strategy for the system is a fairly common one: low-cost
Intel-based servers with a load-balancer in front of them, and big
iron for the database.
=for html
<p><b>Figure 1.</b> Server layout</p>
<img src="machine_layout.png" alt="Machine Layout" width="344"
height="472">
Like many commercial packages, we have separate systems for the
front-end web servers (which we call proxy servers) and the
application servers that generate the dynamic content. Both the proxy
servers and the application servers are load-balanced using dedicated
hardware from f5 Networks.
We chose to run Linux on our proxy and application servers, a common
platform for mod_perl sites. The ease of remote administration under
Linux made the clustered approach possible. Linux also provided solid
security features and automated build capabilities to help with adding
new servers.
The database servers were IBM NUMA-Q machines, which ran on the
DYNIX/ptx operating system..
=head2 Proxy Servers
The proxy servers ran a slim build of Apache, without mod_perl. They
have several standard Apache modules installed, in addition to our own
customized version of mod_session, which assigned session cookies.
Because the processes were so small, we could run up to 400 Apache
children per machine. These servers handled all image requests
themselves, and passed page requests on to the application
servers. They communicated with the app servers using standard HTTP
requests, and cached the page results when appropriate headers are
sent from the app servers. The cached pages were stored on a shared
NFS partition of a Network Appliance filer. Serving pages from the
cache was nearly as fast as serving static files.
This kind of reverse-proxy setup is a commonly recommended approach
when working with mod_perl, since it uses the lightweight proxy
processes to send out the content to clients (who may be on slow
connections) and frees the resource-intensive mod_perl processes to
move on to the next request. For more information on why this
configuration is helpful, see the L<strategy section in the users
guide|guide::strategy>.
=for html
<p><b>Figure 2.</b> Proxy Server Setup</p>
<img src="proxy_servers.png" alt="Proxy Server Setup" width="356"
height="211">
=head2 Application Servers
The application servers ran mod_perl, and very little else. They had
a local cache for Perl objects, using Berkeley DB. The web
applications ran there, and shared resources like HTML templates were
mounted over NFS from the NetApp filer. Because they did the heavy
lifting in this setup, these machines were somewhat beefy, with dual
CPUs and 1GB of RAM each.
=for html
<p><b>Figure 3.</b> Application servers layout</p>
<img src="app_servers.png" alt="Application servers layout"
width="444" height="387">
=head2 Search servers
There was a third set of machines dedicated to handling searches.
Since searching was such a large percentage of overall traffic, it was
worthwhile to dedicate resources to it and take the load off the
application servers and database.
The software on these boxes was a multi-threaded daemon which we
developed in-house using C++. The application servers talked to the
search servers using a Perl module. The search daemon accepted a set
of search conditions and returned a sorted list of object IDs of the
products whose data fits those conditions. Then the application
servers looked up the data to display these products from the
database. The search servers knew nothing about HTML or the web
interface.
This approach of finding the IDs with the search server and then
retrieving the object data may sound like a performance hit, but in
practice the object data usually came from the application server's
cache rather than the database. This design allowed us to minimize
the duplicated data between the database and the search servers,
making it easier and faster to refresh the index. It also let us
reuse the same Perl code for retrieving product objects from the
database, regardless of how they were found.
The daemon used a standard inverted word list approach to searching.
The index was periodically built from the relevant data in
Oracle. There are modules on CPAN which implement this approach,
including C<Search::InvertedIndex> and C<DBIx::FullTextSearch>. We
chose to write our own because of the very tight performance
requirements on this part of the system, and because we had an
unusually complex set of sorting rules for the returned IDs.
=for html
<p><b>Figure 4.</b> Search server layout</p>
<img src="search_servers.png" alt="Search server layout"
width="567" height="269">
=head1 Load Balancing and Failover
We took pains to make sure that we would be able to provide load
balancing among nodes of the cluster and fault tolerance in case one
or more nodes failed. The proxy servers were balanced using a random
selection algorithm. A user could end up on a different one on every
request. These servers didn't hold any state information, so the goal
was just to distribute the load evenly.
The application servers used ``sticky'' load balancing. That means
that once a user went to a particular app server, all of her
subsequent requests during that session were also passed to the same
app server. The f5 hardware accomplished this using browser cookies.
Using sticky load balancing on the app servers allowed us to do some
local caching of user data.
The load balancers ran a periodic service check on every server and
removed any servers that failed the check from rotation. When a server
failed, all users that were ``stuck''; to that machine were moved to
another one.
In order to ensure that no data was lost if an app server died, all
updates were written to the database. As a result, user data like the
contents of a shopping cart was preserved even in cases of
catastrophic hardware failure on an app server. This is essential for
a large e-commerce site.
The database had a separate failover system, which we will not go into
here. It followed standard practices recommended by our vendors.
=head1 Code Structure
The code was structured around the classic Model-View-Controller
pattern, originally from SmallTalk and now often applied to web
applications. The MVC pattern is a way of splitting an application's
responsibilities into three distinct layers.
Classes in the Model layer represented business concepts and data,
like products or users. These had an API but no end-user interface.
They knew nothing about HTTP or HTML and could be used in non-web
applications, like cron jobs. They talked to the database and other
data sources, and managed their own persistence.
The Controller layer translated web requests into appropriate actions
on the Model layer. It handled parsing parameters, checking input,
fetching the appropriate Model objects, and calling methods on them.
Then it determined the appropriate View to use and sendt the resulting
HTML to the user.
View objects were really HTML templates. The Controller passed data
from the Model objects to them and they generated a web page. These
were implemented with the Template Toolkit, a powerful templating
system written in Perl. The templates had some basic conditional
statements and looping in them, but only enough to express the
formatting logic. No application control flow was embedded in the
templates.
=for html
<p><b>Figure 5. </b> <a href="code_structure.png">Code
structure and interaction between the layers</a></p>
=head1 Caching
The core of the performance strategy is a multi-tier caching
system. On the application servers, data objects are cached in shared
memory with a backing store on local disk. Applications specify how
long a data object can be out of sync with the database, and all
future accesses during that time are served from the high-speed
cache. This type of cache control is known as "time-to-live." The
local cache is implemented using a I<Berkeley DB> database. Objects
are serialized with the standard C<Storable> module from CPAN.
Data objects are divided into pieces when necessary to provide finer
granularity for expiration times. For example, product inventory is
updated more frequently than other product data. By splitting the
product data up, we can use a short expiration for inventory that
keeps it in tighter sync with the database, while still using a longer
expiration for the less volatile parts of the product data.
The application servers' object caches share product data between them
using the IP Multicast protocol and custom daemons written in C. When
a product is placed in the cache on one server, the data is replicated
to the cache on all other servers. This technique is very successful
because of the high locality of access in product data. During the
2000 Christmas season this cache achieved a 99% hit ratio, thus taking
a large amount of work off the database.
In addition to caching the data objects, entire pages that are not
user-specific, like product detail pages, can be cached. The
application takes the shortest expiration time of the data objects
used in the pages and specifies that to the proxy servers as a page
expiration time, using standard I<Expires> headers. The proxy servers
cache the generated page on a shared NFS partition. Pages served from
this cache have performance close to that of static pages.
To allow for emergency fixes, we added a hook to C<mod_proxy> that
deletes the cached copy of a specified URL. This was used when a page
needed to be changed immediately to fix incorrect information.
An extra advantage of this C<mod_proxy> cache is the automatic
handling of I<If-Modified-Since> requests. We did not need to
implement this ourselves since C<mod_proxy> already provides it.
=for html
<p><b>Figure 6. </b> <a href="proxy_architecture.png">Proxy and Cache
Interaction</a></p>
=head1 Session Tracking
Users are assigned session IDs using HTTP cookies. This is done at the
proxy servers by our customized version of C<mod_session>. Doing it at
the proxy ensures that users accessing cached pages will still get a
session ID assigned. The session ID is simply a key into data stored
on the server-side. User sessions are assigned to an application
server and continue to use that server unless it becomes
unavailable. This is called ``sticky” load balancing. Session
data and other data modified by the user -- such as shopping cart
contents -- is written to both the object cache and the database. The
double write carries a slight performance penalty, but it allows for
fast read access on subsequent requests without going back to the
database. If a server failure causes a user to be moved to a different
application server, the data is simply fetched from the database
again.
=for html
<p><b>Figure 7.</b> Session tracking and caches</p>
<img src="session_tracking.png" alt="Session tracking and caches" width="331"
height="155">
=head1 Security
A large e-commerce site is a popular target for all types of attacks.
When designing such a system, you have to assume that you will be
attacked and build with security in mind, at the application level as
well as the machine level.
The main rule of thumb is ``don't trust the client!'' User-specific
data sent to the client is protected using multiple levels of
encryption. SSL keeps sensitive data exchanges private from anyone
snooping on network traffic. To prevent ``session hijacking'' (when
someone tampers with their session ID in order to gain access to
another user's session), we include a Message Authentication Code
(MAC) as part of the session cookie. This is generated using the
standard C<Digest::SHA1> module from CPAN, with a seed phrase known
only to our servers. By running the ID from the session cookie through
this MAC algorithm we can verify that the data being presented was
generated by us and not tampered with.
In situations where we need to include some state information in an
HTML form or URL and don't want it to be obvious to the user, we use
the CPAN C<Crypt::> modules to encrypt and decrypt it. The
C<Crypt::CBC> module is a good place to start.
To protect against simple overload attacks, when someone uses a
program to send high volumes of requests at our servers hoping to make
them unavailable to customers, access to the application servers is
controlled by a throttling program. The code is based on some work by
Randal Schwartz in his C<Stonehenge::Throttle> module. Accesses for
each user are tracked in compact logs written to an NFS partition. The
program enforces limits on how many requests a user can make within a
certain period of time.
For more information on web security concerns including the use of
MAC, encryption, and overload prevention, we recommend looking at the
books I<CGI Programming with Perl, 2nd Edition> and I<Writing Apache
Modules with Perl and C>, both from O'Reilly.
=head1 Exception Handling
When planning this system, we considered using Java as the
implementation language. We decided to go with Perl, but we really
missed Java's nice exception handling features. Luckily, Graham Barr's
Error module from CPAN supplies similar capabilities in Perl.
Perl already has support for trapping runtime errors and passing
exception objects, but the Error module adds some nice syntactic
sugar. The following code sample is typical of how we used the
module:
try {
do_some_stuff();
} catch My::Exception with {
my $E = shift;
handle_exception($E);
};
The module allows you to create your own exception classes and trap
for specific types of exceptions.
One nice benefit of this is the way it works with C<DBI>. If you turn
on C<DBI>'s I<RaiseError> flag and use try blocks in places where you
want to trap exceptions, the C<Error> module can turn C<DBI> errors
into simple C<Error> objects.
try {
$sth->execute();
} catch Error with {
# roll back and recover
$dbh->rollback();
# etc.
};
This code shows a condition where an error would indicate that we
should roll back a database transaction. In practice, most C<DBI>
errors indicate something unexpected happened with the database and
the current action can't continue. Those exceptions are allowed to
propagate up to a top-level C<try{}> block that encloses the whole
request. When errors are caught there, we log a stacktrace and send a
friendly error page back to the user.
=head1 Templates
Both the HTML and the formatting logic for merging application data
into it is stored in the templates. They use a CPAN module called
I<Template Toolkit>, which provides a simple but powerful syntax for
accessing the Perl data structures passed to them by the application.
In addition to basics like looping and conditional statements, it
provides extensive support for modularization, allowing the use of
includes and macros to simplify template maintenance and avoid
redundancy.
We found I<Template Toolkit> to be an invaluable tool on this project.
Our HTML coders picked it up very quickly and were able to do nearly
all of the templating work without help from the Perl coders. We
supplied them with documentation of what data would be passed to each
template and they did the rest. If you have never experienced the joy
of telling a project manager that the HTML team can handle his
requested changes without any help from you, you are seriously missing
out!
I<Template Toolkit> compiles templates into Perl bytecode and caches
them in memory to improve efficiency. When template files change on
disk they are picked up and re-compiled. This is similar to how other
C<mod_perl> systems like Mason and C<Apache::Registry> work.
By varying the template search path, we made it possible to assign
templates to particular sections of the site, allowing a customized
look and feel for specific areas. For example, the page header
template in the bookstore section of the site can be different from
the one in the video game store section. It is even possible to serve
the same data with a different appearance in different parts of the
site, allowing for co-branding of content.
This is a sample of what a basic loop looks like when coded in
I<Template Toolkit>:
[% FOREACH item = cart.items %]
name: [% item.name %]
price: [% item.price %]
[% END %]
=head1 Controller Example
Let's walk through a simple Hello World example that illustrates how
the Model-View-Controller pattern is used in our code. We'll start with
the controller code.
package ESF::Control::Hello;
use strict;
use ESF::Control;
@ESF::Control::Hello::ISA = qw(ESF::Control);
use ESF::Util;
sub handler {
### do some setup work
my $class = shift;
my $apr = ESF::Util->get_request();
### instantiate the model
my $name = $apr->param('name');
# we create a new Model::Hello object.
my $hello = ESF::Model::Hello-E<gt>new(NAME =E<gt> $name);
### send out the view
my $view_data{'hello'} = $hello->view();
# the process_template() method is inherited
# from the ESF::Control base class
$class->process_template(
TEMPLATE => 'hello.html',
DATA => \%view_data);
}
In addition to the things you see here, there are a few interesting
details about the C<ESF::Control> base class. All requests are
dispatched to the C<ESF::Control-E<gt>run()> method first, which wraps
them in a C<try{}> block before calling the appropriate C<handler()>
method. It also provides the C<process_template()> method, which runs
C<Template Toolkit> and then sends out the results with appropriate
HTTP headers. If the Controller specifies it, the headers can include
C<Last-Modified> and C<Expires>, for control of page caching by the
proxy servers.
Now let's look at the corresponding Model code.
package ESF::Model::Hello;
use strict;
sub new {
my $class = shift;
my %args = @_;
my $self = bless {}, $class;
$self{'name'} = $args{'NAME'} || 'World';
return $self;
}
sub view {
# the object itself will work for the view
return shift;
}
This is a very simple Model object. Most Model objects would have some
database and cache interaction. They would include a C<load()> method
which accepts an ID and loads the appropriate object state from the
database. Model objects that can be modified by the application would
also include a C<save()> method.
Note that because of Perl's flexible OO style, it is not necessary to
call C<new()> when loading an object from the database. The C<load()>
and C<new()> methods can both be constructors for use in different
circumstances, both returning a blessed reference.
The C<load()> method typically handles cache management as well as
database access. Here's some pseudo-code showing a typical C<load()>
method:
sub load {
my $class = shift;
my %args = @_;
my $id = $args{'ID'};
my $self;
unless ($self = _fetch_from_cache($id)) {
$self = _fetch_from_database($id);
$self->_store_in_cache();
}
return $self;
}
The save method would use the same approach in reverse, saving first
to the cache and then to the database.
One final thing to notice about our Model class is the C<view()>
method. This method exists to give the object an opportunity to
shuffle it's data around or create a separate data structure that is
easier for use with a template. This can be used to hide a complex
implementation from the template coders. For example, remember the
partitioning of the product inventory data that we did to allow for
separate cache expiration times? The product Model object is really a
faE<ccedil>ade for several underlying implementation objects, but the
C<view()> method on that class consolidates the data for use by the
templates.
To finish off our Hello World example, we need a template to render
the view. This one will do the job:
<html>
<title>Hello, My Oyster</title>
<body>
[% PROCESS header.html %]
Hello [% hello.name %]!
[% PROCESS footer.html %]
</body>
</html>
=head1 Performance Tuning
Since Perl code executes so quickly under C<mod_perl>, the performance
bottleneck is usually at the database. We applied all the documented
tricks for improving C<DBD::Oracle> performance. We used bind
variables, C<prepare_cached()>, C<Apache::DBI>, and adjustments to the
C<RowCache> buffer size.
The big win of course is avoiding going to the database in the first
place. The caching work we did had a huge impact on performance.
Fetching product data from the I<Berkeley DB> cache was about ten
times faster than fetching it from the database. Serving a product
page from the proxy cache was about ten times faster than generating
it on the application server from cached data. Clearly the site would
never have survived under heavy load without the caching.
Partitioning the data objects was also a big win. We identified
several different subsets of product data that could be loaded and
cached independently. When an application needed product data, it
could specify which subset was required and skip loading the
unnecessary data from the database.
Another standard performance technique we followed was avoiding
unnecessary object creation. The C<Template> object is created the
first time it's used and then cached for the life of the Apache
process. Socket connections to search servers are cached in a way
similar to what C<Apache::DBI> does for database
connections. Resources that are used frequently within the scope of a
request, such as database handles and session objects, were cached in
C<mod_perl>'s C<$r-E<gt>pnotes()> until the end of the request.
=head1 Trap: Nested Exceptions
When trying out a new technology like the C<Error> module, there are
bound to be some things to watch out for. We found a certain code
structure that causes a memory leak every time it is executed. It
involves nested C<try{}> blocks, and looks like this:
my $foo;
try {
# some stuff...
try {
$foo++;
# more stuff...
} catch Error with {
# handle error
};
} catch Error with {
# handle other error
};
It's not Graham Barr's fault that this leaks; it is simply a
by-product of the fact that the C<try> and C<catch> keywords are
implemented using anonymous subroutines. This code is equivalent to
the following:
my $foo;
$subref1 = sub {
$subref2 = sub {
$foo++;
};
};
This nested subroutine creates a closure for C<$foo> and will make a
new copy of the variable every time it is executed. The situation is
easy to avoid once you know to watch out for it.
=head2 Berkeley DB
One of the big wins in our architecture was the use of I<Berkeley DB>.
Since most people are not familiar with it's more advanced features,
we'll give a brief overview here.
The C<DB_File> module is part of the standard Perl distribution.
However, it only supports the interface of I<Berkeley DB> version
1.85, and doesn't include the interesting features of later
releases. To get those, you'll need the C<BerkeleyDB.pm> module,
available from CPAN. This module can be tricky to build, but
comprehensive instructions are included.
Newer versions of I<Berkeley DB> offer many features that help
performance in a C<mod_perl> environment. To begin with, database
files can be opened once at the start of the program and kept open,
rather than opened and closed on every request. I<Berkeley DB> will
use a shared memory buffer to improve data access speed for all
processes using the database. Concurrent access is directly supported
with locking handled for you by the database. This is a huge win over
C<DB_File>, which requires you to do your own locking. Locks can be at
a database level, or at a memory page level to allow multiple
simultaneous writers. Transactions with rollback capability are also
supported.
This all sounds too good to be true, but there are some downsides. The
documentation is somewhat sparse, and you will probably need to refer
to the C API if you need to understand how to do anything complicated.
A more serious problem is database corruption. When an Apache process
using I<Berkeley DB> dies from a hard kill or a segfault, it can
corrupt the database. A corrupted database will sometimes cause
subsequent opening attempts to hang. According to the people we talked
to at Sleepycat Software (which provides commercial support for
I<Berkeley DB>), this can happen even with the transactional mode of
operation. They are working on a way to fix the problem. In our case,
none of the data stored in the cache was essential for operation so we
were able to simply clear it out when restarting an application
server.
Another thing to watch out for is deadlocks. If you use the page-level
locking option, you have to handle deadlocks. There is a daemon
included in the distribution that will watch for deadlocks and fix
them, or you can handle them yourself using the C API.
After trying a few different things, we recommend that you use
database-level locking. It's much simpler, and cured our problems. We
didn't see any significant performance hit from switching to this mode
of locking. The one thing you need to watch out for when using
exclusive database level write locks are long operations with cursors
that tie up the database. We split up some of our operations into
multiple writes in order to avoid this problem.
If you have a good C coder on your team, you may want to try the
alternate approach that we finally ended up with. You can write your
own daemon around I<Berkeley DB> and use it in a client/server style
over Unix sockets. This allows you to catch signals and ensure a safe
shutdown. You can also write your own deadlock handling code this way.
=head1 Valuable Tools
If you plan to do any serious Perl development, you should really take
the time to become familiar with some of the available development
tools. The debugger in particular is a lifesaver, and it works with
C<mod_perl>. There is a profiler called C<Devel::DProf>, which also
works with C<mod_perl>. It's definitely the place to start when
performance tuning your application.
We found the ability to run our complete system on individual's
workstations to be extremely useful. Everyone could develop on his own
machine, and coordinate changes using I<CVS> source control.
For object modeling and design, we used the open source C<Dia> program
and I<Rational Rose>. Both support working with UML and are great for
generating pretty class diagrams for your cubicle walls.
=head1 Do Try This at Home
Since we started this project, a number of development frameworks that
offer support for this kind of architecture have come out. We don't
have direct experience using these, but they have a similar design and
may prove useful to you if you want to take an MVC approach with your
system.
C<Apache::PageKit> is a C<mod_perl> module available from CPAN which
provides a basic MVC structure for web applications. It uses the
C<HTML::Template> module for building views.
I<OpenInteract> is a recently released web application framework in
Perl, which works together with the persistence layer C<SPOPS>. Both
are available from CPAN.
The I<Application Toolkit> from Extropia is a comprehensive set of
Perl classes for building web apps. It has excellent documentation and
takes good advantage of existing CPAN modules. You can find it on
http://www.extropia.com/.
If you want a ready-to-use cache module, take a look at the Perl-cache
project on http://sourceforge.net/. This is the next generation of the
popular C<File::Cache> module.
The Java world has many options as well. The I<Struts> framework, part
of the I<Jakarta> project, is a good open source choice. There are
also commercial products from several vendors that follow this sort of
design. Top contenders include I<ATG Dynamo>, I<BEA WebLogic>, and
I<IBM WebSphere>.
=head1 An Open Source Success Story
By building on the open source software and community, we were able to
create a top-tier web site with a minimum of cost and effort. The
system we ended up with is scalable to huge amounts of traffic. It
runs on mostly commodity hardware making it easy to grow when the need
arises. Perhaps best of all, it provided tremendous learning
opportunities for our developers, and made us a part of the larger
development community.
We've contributed patches from our work back to various open source
projects, and provided help on mailing lists. We'd like to take this
opportunity to officially thank the open source developers who
contributed to projects mentioned here. Without them, this would not
have been possible. We also have to thank the hardworking web
developers at eToys. The store may be closed, but the talent that
built it lives on.
=head1 Maintainers
The maintainer is the person(s) you should contact with updates,
corrections and patches.
Per Einar Ellefsen E<lt>per.einar (at) skynet.beE<gt>
=head1 Authors
=over
=item * Bill Hilf E<lt>bill (at) hilfworks.comE<gt>
=item * Perrin Harkins E<lt>perrin (at) elem.comE<gt>
=back
Only the major authors are listed above. For contributors see the
Changes file.
=cut
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/machine_layout.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/proxy_architecture.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/proxy_servers.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/search_servers.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/apps/scale_etoys/session_tracking.png
<<Binary file>>
1.1
modperl-docs/src/docs/tutorials/client/browserbugs/browserbugs.pod
Index: browserbugs.pod
===================================================================
=head1 NAME
Workarounds for some known bugs in browsers.
=head1 Description
Unfortunately for web programmers, browser bugs are not uncommon, and
sometimes we have to deal with them; refer to this chapter for some
known bugs and how you can work around them.
=head1 Preventing QUERY_STRING from getting corrupted because of &entity key
names
In a URL which contains a query string, if the string has multiple
parts separated by ampersands and it contains a key named "reg", for
example C<http://example.com/foo.pl?foo=bar®=foobar>, then some
browsers will interpret C<®> as an SGML entity and encode it as
C<®>. This will result in a corrupted C<QUERY_STRING>. If you
encounter this problem, then either you should avoid using such keys
or you should separate parameter pairs with C<;> instead of C<&>.
C<CGI.pm>, C<Apache::Request> and C<$r-E<gt>args()> support a semicolon
instead of an ampersand as a separator. So your URI should look like
this: C<http://example.com/foo.pl?foo=bar;reg=foobar>.
Note that this is only an issue when you are building your own URLs
with query strings. It is not a problem when the URL is the result
of submitting a form because the browsers I<have> to get that right.
=head1 IE 4.x does not re-post data to a non-port-80 URL
One problem with publishing 8080 port numbers (or so I have been
told) is that IE 4.x has a bug when re-posting data to a non-port-80
URL. It drops the port designator and uses port 80 anyway.
See L<Publishing Port Numbers other than
80|guide::config/Publishing_Port_Numbers_other_than_80>.
=head1 Internet Explorer disregards your ErrorDocuments
Many users stumble upon a common problem related to MS Internet
Explorer: if your error response, such as when using C<ErrorDocument
500> or C<$r-E<gt>custom_response>, is too short (which might often be
the case because you aren't very inspired when writing error
messages), Internet Explorer completely disregards it and replaces it
with its own standard error page, even though everything has been sent
correctly by the server and received by the browser.
The solution to this is quite simple: your content needs to be at
least 512 bytes. Microsoft describes some solutions to this I<problem>
here: http://support.microsoft.com/support/kb/articles/Q294/8/07.ASP
. The easiest solution under Perl is to do something like this:
# write your HTML headers
print "<!-- ", "_" x 513, " -->";
# write out the rest of your HTML
Effectively, your content will be long enough, but the user won't
notice any additional content. If you're doing this with static pages,
just insert a long enough comment inside your file to make it large
enough, which will have the same effect.
=head1 Maintainers
Maintainer is the person(s) you should contact with updates,
corrections and patches.
=over
=item *
Stas Bekman E<lt>stas (at) stason.orgE<gt>
=back
=head1 Authors
=over
=item *
Stas Bekman E<lt>stas (at) stason.orgE<gt>
=back
Only the major authors are listed above. For contributors see the
Changes file.
=cut
1.1
modperl-docs/src/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.pod
Index: mod_perl_tricks.pod
===================================================================
=head1 NAME
Cute Tricks With Perl and Apache
=head1 Description
Perl and Apache play very well together, both for administration and
coding. However, adding mod_perl to the mix creates a heaven for an
administrator/programmer wanting to do cool things in no time!
This tutorial begins a collection of CGI scripts that illustrate the
four basic types of CGI scripting: dynamic documents, document
filtering, and URL redirection. It also shows a few tricks that you
might not have run into -- or even thought were possible with CGI.
Then, we move to look at different uses of Perl to handle typical
administrative tasks. Finally, we continue with the next step beyond
CGI scripting: the creation of high performance Apache modules with
the mod_perl API.
=head1 Part I: Tricks with CGI.pm
C<CGI.pm> is the long-favoured module for CGI scripting, and, as
mod_perl can run CGI scripts (mostly) unaltered, also provides
significant advantages for mod_perl programmers. Let's look at some of
the more interesting uses of this module in web programming.
=head2 Dynamic Documents
The most familiar use of CGI is to create documents on the fly. They
can be simple documents, or get incredibly baroque. We won't venture
much past the early baroque.
=head3 Making HTML look beautiful
E<lt>IE<gt> E<lt>hateE<gt> E<lt>HTMLE<gt> E<lt>becauseE<gt>
E<lt>it'sE<gt> E<lt>uglyE<gt> E<lt>andE<gt> E<lt>hasE<gt>
E<lt>tooE<gt> E<lt>manyE<gt> E<lt>[EMAIL PROTECTED]&E<gt> E<lt>angleE<gt>
E<lt>bracketsE<gt>. With CGI.pm it's almost good to look at. Script
I.1.1 shows what a nested list looks like with CGI.pm.
Script I.1.1: vegetables1.pl
----------------------------
#!/usr/bin/perl
# Script: vegetables1.pl
use CGI::Pretty ':standard';
print header,
start_html('Vegetables'),
h1('Eat Your Vegetables'),
ol(
li('peas'),
li('broccoli'),
li('cabbage'),
li('peppers',
ul(
li('red'),
li('yellow'),
li('green')
)
),
li('kolrabi'),
li('radishes')
),
hr,
end_html;
=head3 Making HTML concise
But we can do even better than that because CGI.pm lets you collapse
repeating tags by passing array references to its functions. Script
1.2 saves some typing, and in so doing, puts off the onset of RSI by
months or years!
Script I.1.2: vegetables2.pl
----------------------------
#!/usr/bin/perl
# Script: vegetables2.pl
use CGI ':standard';
print header,
start_html('Vegetables'),
h1('Eat Your Vegetables'),
ol(
li(['peas',
'broccoli',
'cabbage',
'peppers' .
ul(['red','yellow','green']),
'kolrabi',
'radishes'
),
hr,
end_html;
Or how about this one?
Script I.1.3: vegetables3.pl
----------------------------
#!/usr/bin/perl
# Script: vegetables3.pl
use CGI::Pretty qw/:standard :html3/;
print header,
start_html('Vegetables'),
h1('Vegetables are for the Strong'),
table({-border=>undef},
caption(strong('When Should You Eat Your Vegetables?')),
Tr({-align=>CENTER,-valign=>TOP},
[
th(['','Breakfast','Lunch','Dinner']),
th('Tomatoes').td(['no','yes','yes']),
th('Broccoli').td(['no','no','yes']),
th('Onions').td(['yes','yes','yes'])
]
)
),
end_html;
=head3 Making Interactive Forms
Of course you mostly want to use CGI to create interactive forms. No
problem! CGI.pm has a full set of functions for both generating the
form and reading its contents once submitted. Script I.1.4 creates a
row of radio buttons labeled with various colors. When the user
selects a button and submits the form, the page redraws itself with
the selected background color. Psychedelic!
Script I.1.4: customizable.pl
-----------------------------
#!/usr/bin/perl
# script: customizable.pl
use CGI::Pretty qw/:standard/;
my $color = param('color') || 'white';
print header,
start_html({-bgcolor=>$color},'Customizable Page'),
h1('Customizable Page'),
"Set this page's background color to:",br,
start_form,
radio_group(-name=>'color',
-value=>['white','red','green','black',
'blue','silver','cyan'],
-cols=>2),
submit(-name=>'Set Background'),
end_form,
p,
hr,
end_html;
=head2 Making Stateful Forms
Many real Web applications are more than a single page. Some may span
multiple pages and fill-out forms. When the user goes from one page
to the next, you've got to save the state of the previous page
somewhere. A convenient and cheap place to put state information is
in hidden fields in the form itself. Script I.2.1 is an example of a
loan application with a total of five separate pages. Forward and
back buttons allows the user to navigate between pages. The script
remembers all the pages and summarizes them up at the end.
Script I.2.1: loan.pl
---------------------
#!/usr/local/bin/perl
# script: loan.pl
use CGI qw/:standard :html3/;
# this defines the contents of the fill out forms
# on each page.
my @PAGES = ('Personal
Information','References','Assets','Review','Confirmation');
my %FIELDS = ('Personal Information' =>
['Name','Address','Telephone','Fax'],
'References' => ['Personal Reference 1','Personal
Reference 2'],
'Assets' => ['Savings Account','Home','Car']
);
my %ALL_FIELDS = ();
# accumulate the field names into %ALL_FIELDS;
foreach (values %FIELDS) {
grep($ALL_FIELDS{$_}++, @$_);
}
# figure out what page we're on and where we're heading.
my $current_page = calculate_page(param('page'),param('go'));
my $page_name = $PAGES[$current_page];
print_header($page_name);
print_form($current_page) if $FIELDS{$page_name};
print_review($current_page) if $page_name eq 'Review';
print_confirmation($current_page) if $page_name eq 'Confirmation';
print end_html;
# CALCULATE THE CURRENT PAGE
sub calculate_page {
my ($prev, $dir) = @_;
return 0 if $prev eq ''; # start with first page
return $prev + 1 if $dir eq 'Submit Application';
return $prev + 1 if $dir eq 'Next Page';
return $prev - 1 if $dir eq 'Previous Page';
}
# PRINT HTTP AND HTML HEADERS
sub print_header {
my $page_name = shift;
print header,
start_html("Your Friendly Family Loan Center"),
h1("Your Friendly Family Loan Center"),
h2($page_name);
}
# PRINT ONE OF THE QUESTIONNAIRE PAGES
sub print_form {
my $current_page = shift;
print "Please fill out the form completely and accurately.",
start_form,
hr;
draw_form(@{$FIELDS{$page_name}});
print hr;
print submit(-name=>'go',-value=>'Previous Page')
if $current_page > 0;
print submit(-name=>'go',-value=>'Next Page'),
hidden(-name=>'page',-value=>$current_page,-override=>1),
end_form;
}
# PRINT THE REVIEW PAGE
sub print_review {
my $current_page = shift;
print "Please review this information carefully before submitting it. ",
start_form;
my (@rows);
foreach $page ('Personal Information','References','Assets') {
push(@rows,th({-align=>LEFT},em($page)));
foreach $field (@{$FIELDS{$page}}) {
push(@rows,
TR(th({-align=>LEFT},$field),
td(param($field)))
);
print hidden(-name=>$field);
}
}
print table({-border=>1},caption($page),@rows),
hidden(-name=>'page',-value=>$current_page,-override=>1),
submit(-name=>'go',-value=>'Previous Page'),
submit(-name=>'go',-value=>'Submit Application'),
end_form;
}
# PRINT THE CONFIRMATION PAGE
sub print_confirmation {
print "Thank you. A loan officer will be contacting you shortly.",
p,
a({-href=>'../source.html'},'Code examples');
}
# CREATE A GENERIC QUESTIONNAIRE
sub draw_form {
my (@fields) = @_;
my (%fields);
grep ($fields{$_}++, @fields);
my (@hidden_fields) = grep(!$fields{$_}, keys %ALL_FIELDS);
my (@rows);
foreach (@fields) {
push(@rows,
TR(th({-align=>LEFT},$_),
td(textfield(-name=>$_,-size=>50))
)
);
}
print table(@rows);
foreach (@hidden_fields) {
print hidden(-name=>$_);
}
}
=head3 Keeping State with Cookies
If you want to maintain state even if the user quits the browser and
comes back again, you can use cookies. Script I.2.2 records the
user's name and color scheme preferences and recreates the page the
way the user likes up to 30 days from the time the user last used the
script.
Script I.2.2: preferences.pl
----------------------------
#!/usr/local/bin/perl
# file: preferences.pl
use CGI qw(:standard :html3);
# Some constants to use in our form.
my @colors = qw/aqua black blue fuschia gray green lime maroon navy olive
purple red silver teal white yellow/;
my @sizes=("<default>",1..7);
# recover the "preferences" cookie.
my %preferences = cookie('preferences');
# If the user wants to change the background color or her
# name, they will appear among our CGI parameters.
foreach ('text','background','name','size') {
$preferences{$_} = param($_) || $preferences{$_};
}
# Set some defaults
$preferences{'background'} ||= 'silver';
$preferences{'text'} ||= 'black';
# Refresh the cookie so that it doesn't expire.
my $the_cookie = cookie(-name=>'preferences',
-value=>\%preferences,
-path=>'/',
-expires=>'+30d');
print header(-cookie=>$the_cookie);
# Adjust the title to incorporate the user's name, if provided.
$title = $preferences{'name'} ?
"Welcome back, $preferences{name}!" : "Customizable Page";
# Create the HTML page. We use several of the HTML 3.2
# extended tags to control the background color and the
# font size. It's safe to use these features because
# cookies don't work anywhere else anyway.
print start_html(-title=>$title,
-bgcolor=>$preferences{'background'},
-text=>$preferences{'text'}
);
print basefont({-size=>$preferences{size}}) if $preferences{'size'} > 0;
print h1($title);
# Create the form
print hr,
start_form,
"Your first name: ",
textfield(-name=>'name',
-default=>$preferences{'name'},
-size=>30),br,
table(
TR(
td("Preferred"),
td("Page color:"),
td(popup_menu(-name=>'background',
-values=>[EMAIL PROTECTED],
-default=>$preferences{'background'})
),
),
TR(
td(''),
td("Text color:"),
td(popup_menu(-name=>'text',
-values=>[EMAIL PROTECTED],
-default=>$preferences{'text'})
)
),
TR(
td(''),
td("Font size:"),
td(popup_menu(-name=>'size',
-values=>[EMAIL PROTECTED],
-default=>$preferences{'size'})
)
)
),
submit(-label=>'Set preferences'),
end_form,
hr,
end_html;
=head2 Creating Non-HTML Types
CGI can do more than just produce HTML documents. It can produce any
type of document that you can output with Perl. This includes GIFs,
Postscript files, sounds or whatever.
Script I.3.1 creates a clickable image map of a colored circle inside
a square. The script is responsible both for generating the map and
making the image (using the GD.pm library). It also creates a
fill-out form that lets the user change the size and color of the image!
Script I.3.1: circle.pl
-----------------------
#!/usr/local/bin/perl
# script: circle.pl
use GD;
use CGI qw/:standard :imagemap/;
use constant RECTSIZE => 100;
use constant CIRCLE_RADIUS => 40;
my %COLORS = (
'white' => [255,255,255],
'red' => [255,0,0],
'green' => [0,255,0],
'blue' => [0,0,255],
'black' => [0,0,0],
'bisque'=> [255,228,196],
'papaya whip' => [255,239,213],
'sienna' => [160,82,45]
);
my $draw = param('draw');
my $circle_color = param('color') || 'bisque';
my $mag = param('magnification') || 1;
if ($draw) {
draw_image();
} else {
make_page();
}
sub draw_image {
# create a new image
my $im = new GD::Image(RECTSIZE*$mag,RECTSIZE*$mag);
# allocate some colors
my $white = $im->colorAllocate(@{$COLORS{'white'}});
my $black = $im->colorAllocate(@{$COLORS{'black'}});
my $circlecolor = $im->colorAllocate(@{$COLORS{$circle_color}});
# make the background transparent and interlaced
$im->transparent($white);
$im->interlaced('true');
# Put a black frame around the picture
$im->rectangle(0,0,RECTSIZE*$mag-1,RECTSIZE*$mag-1,$black);
# Draw the circle
$im->arc(RECTSIZE*$mag/2,RECTSIZE*$mag/2,
CIRCLE_RADIUS*$mag*2,
CIRCLE_RADIUS*$mag*2,
0,360,$black);
# And fill it with circlecolor
$im->fill(RECTSIZE*$mag/2,RECTSIZE*$mag/2,$circlecolor);
# Convert the image to GIF and print it
print header('image/gif'),$im->gif;
}
sub make_page {
print header(),
start_html(-title=>'Feeling Circular',-bgcolor=>'white'),
h1('A Circle is as a Circle Does'),
start_form,
"Magnification: ",radio_group(-name=>'magnification',-values=>[1..4]),br,
"Color: ",popup_menu(-name=>'color',-values=>[sort keys %COLORS]),
submit(-value=>'Change'),
end_form;
print em(param('message') || 'click in the drawing' );
my $url = url(-relative=>1,-query_string=>1);
$url .= '?' unless param();
$url .= '&draw=1';
print p(
img({-src=>$url,
-align=>'LEFT',
-usemap=>'#map',
-border=>0}));
print Map({-name=>'map'},
Area({-shape=>'CIRCLE',
-href=>param(-name=>'message',-value=>"You clicked in
the circle")
&& url(-relative=>1,-query_string=>1),
-coords=>join(',',RECTSIZE*$mag/2,RECTSIZE*$mag/2,CIRCLE_RADIUS*$mag),
-alt=>'Circle'}),
Area({-shape=>'RECT',
-href=>param(-name=>'message',-value=>"You clicked in
the square")
&& url(-relative=>1,-query_string=>1),
-coords=>join(',',0,0,RECTSIZE*$mag,RECTSIZE*$mag),
-alt=>'Square'}));
print end_html;
}
Script I.3.2 creates a GIF89a animation. First it creates a set of
simple GIFs, then uses the I<combine> program (part of the ImageMagick
package) to combine them together into an animation.
I'm not a good animator, so I can't do anything fancy. But you can!
Script I.3.2: animate.pl
------------------------
#!/usr/local/bin/perl
# script: animated.pl
use GD;
use File::Path;
use constant START => 80;
use constant END => 200;
use constant STEP => 10;
use constant COMBINE => '/usr/local/bin/convert';
my @COMBINE_OPTIONS = (-delay => 5,
-loop => 10000);
my @COLORS = ([240,240,240],
[220,220,220],
[200,200,200],
[180,180,180],
[160,160,160],
[140,140,140],
[150,120,120],
[160,100,100],
[170,80,80],
[180,60,60],
[190,40,40],
[200,20,20],
[210,0,0]);
@COLORS = (@COLORS,reverse(@COLORS));
my @FILES = ();
my $dir = create_temporary_directory();
my $index = 0;
for (my $r = START; $r <= END; $r+=STEP) {
draw($r,$index,$dir);
$index++;
}
for (my $r = END; $r > START; $r-=STEP) {
draw($r,$index,$dir);
$index++;
}
# emit the GIF89a
$| = 1;
print "Content-type: image/gif\n\n";
system COMBINE,@COMBINE_OPTIONS,@FILES,"gif:-";
rmtree([$dir],0,1);
sub draw {
my ($r,$color_index,$dir) = @_;
my $im = new GD::Image(END,END);
my $white = $im->colorAllocate(255,255,255);
my $black = $im->colorAllocate(0,0,0);
my $color = $im->colorAllocate(@{$COLORS[$color_index % @COLORS]});
$im->rectangle(0,0,END,END,$white);
$im->arc(END/2,END/2,$r,$r,0,360,$black);
$im->fill(END/2,END/2,$color);
my $file = sprintf("%s/picture.%02d.gif",$dir,$color_index);
open (OUT,">$file") || die "couldn't create $file: $!";
print OUT $im->gif;
close OUT;
push(@FILES,$file);
}
sub create_temporary_directory {
my $basename = "/usr/tmp/animate$$";
my $counter=0;
while ($counter < 100) {
my $try = sprintf("$basename.%04d",$counter);
next if -e $try;
return $try if mkdir $try,0700;
} continue { $counter++; }
die "Couldn't make a temporary directory";
}
=head2 Document Translation
Did you know that you can use a CGI script to translate other
documents on the fly? No s**t! Script I.4.1 is a script that
intercepts all four-letter words in text documents and stars out the
naughty bits. The document itself is specified using additional path
information. We're a bit over-literal about what a four-letter word
is, but what's the fun if you can't be extravagant?
Script I.4.1: naughty.pl
#!/usr/local/bin/perl
# Script: naughty.pl
use CGI ':standard';
my $file = path_translated() ||
die "must be called with additional path info";
open (FILE,$file) || die "Can't open $file: $!\n";
print header('text/plain');
while (<FILE>) {
s/\b(\w)\w{2}(\w)\b/$1**$2/g;
print;
}
close FILE;
4.1 won't work on HTML files because the HTML tags will get starred
out too. If you find it a little limiting to work only on plain-text
files, script I.4.2 uses LWP's HTML parsing functions to modify just
the text part of an HTML document without touching the tags. The
script's a little awkward because we have to guess the type of file
from the extension, and I<redirect> when we're dealing with a non-HTML
file. We can do better with I<mod_perl>.
Script I.4.2: naughty2.pl
-------------------------
#!/usr/local/bin/perl
# Script: naughty2.pl
package HTML::Parser::FixNaughty;
require HTML::Parser;
@HTML::Parser::FixNaughty::ISA = 'HTML::Parser';
sub start {
my ($self,$tag,$attr,$attrseq,$origtext) = @_;
print $origtext;
}
sub end {
my ($self,$tag) = @_;
print "</$tag>";
}
sub text {
my ($self,$text) = @_;
$text =~ s/\b(\w)\w{2}(\w)\b/$1**$2/g;
print $text;
}
package main;
use CGI qw/header path_info redirect path_translated/;
my $file = path_translated() ||
die "must be called with additional path info";
$file .= "index.html" if $file =~ m!/$!;
unless ($file =~ /\.html?$/) {
print redirect(path_info());
exit 0;
}
my $parser = new HTML::Parser::FixNaughty;
print header();
$parser->parse_file($file);
A cleaner way to do this is to make this into an Apache Handler
running under mod_perl. Let's look at C<Apache::FixNaughty>:
file:Apache/FixNaughty.pm
-------------------------
# prefefine the HTML parser that we use afterwards
package HTML::Parser::FixNaughty;
require HTML::Parser;
@HTML::Parser::FixNaughty::ISA = 'HTML::Parser';
sub start {
my ($self,$tag,$attr,$attrseq,$origtext) = @_;
print $origtext;
}
sub end {
my ($self,$tag) = @_;
print "</$tag>";
}
sub text {
my ($self,$text) = @_;
$text =~ s/\b(\w)\w{2}(\w)\b/$1**$2/g;
print $text;
}
# now for the mod_perl handler
package Apache::FixNaughty;
use Apache::Constants qw/:common/;
use strict;
use warnings;
use CGI qw/header path_info redirect path_translated/;
sub handler {
my $r = shift;
unless(-e $r->finfo) {
$r->log_reason("Can't be found", $r->filename);
return NOT_FOUND;
}
unless ($r->content_type eq 'text/html') {
return DECLINED;
}
my $parser = new HTML::Parser::FixNaughty;
$r->send_http_header('text/html');
$parser->parse_file($file);
return OK;
}
1;
__END__
You'll configure this like so:
Alias /naughty/ /path/to/doc/root/
<Location /naughty>
SetHandler perl-script
PerlHandler Apache::FixNaughty
</Location>
Now, all files being served below the I</naughty> URL will be the same
as those served from your document root, but will be processed and
censured!
=head3 Smart Redirection
There's no need even to create a document with CGI. You can simply
I<redirect> to the URL you want. Script I.4.3 chooses a random
picture from a directory somewhere and displays it. The directory to
pick from is specified as additional path information, as in:
/cgi-bin/random_pict/banners/egregious_advertising
Script I.4.3 random_pict.pl
---------------------------
#!/usr/local/bin/perl
# script: random_pict.pl
use CGI qw/:standard/;
my $PICTURE_PATH = path_translated();
my $PICTURE_URL = path_info();
chdir $PICTURE_PATH
or die "Couldn't chdir to pictures directory: $!";
my @pictures = <*.{jpg,gif}>;
my $lucky_one = $pictures[rand(@pictures)];
die "Failed to pick a picture" unless $lucky_one;
print redirect("$PICTURE_URL/$lucky_one");
Under mod_perl, you would do this (the bigger size is because we're
doing more checks here):
file:Apache/RandPicture.pm
--------------------------
package Apache::RandPicture;
use strict;
use Apache::Constants qw(:common REDIRECT);
use DirHandle ();
sub handler {
my $r = shift;
my $dir_uri = $r->dir_config('PictureDir');
unless ($dir_uri) {
$r->log_reason("No PictureDir configured");
return SERVER_ERROR;
}
$dir_uri .= "/" unless $dir_uri =~ m:/$:;
my $subr = $r->lookup_uri($dir_uri);
my $dir = $subr->filename;
# Get list of images in the directory.
my $dh = DirHandle->new($dir);
unless ($dh) {
$r->log_error("Can't read directory $dir: $!");
return SERVER_ERROR;
}
my @files;
for my $entry ($dh->read) {
# get the file's MIME type
my $rr = $subr->lookup_uri($entry);
my $type = $rr->content_type;
next unless $type =~ m:^image/:;
push @files, $rr->uri;
}
$dh->close;
unless (@files) {
$r->log_error("No image files in directory");
return SERVER_ERROR;
}
my $lucky_one = $files[rand @files];
# internal redirect, so we don't have to go back to the client
$r->internal_redirect($lucky_one);
return REDIRECT;
}
1;
__END__
=head2 File Uploads
Everyone wants to do it. I don't know why. Script I.5.1 shows a
basic script that accepts a file to upload, reads it, and prints out
its length and MIME type. Windows users should read about binmode()
before they try this at home!
Script I.5.1 upload.pl
----------------------
#!/usr/local/bin/perl
#script: upload.pl
use CGI qw/:standard/;
print header,
start_html('file upload'),
h1('file upload');
print_form() unless param;
print_results() if param;
print end_html;
sub print_form {
print start_multipart_form(),
filefield(-name=>'upload',-size=>60),br,
submit(-label=>'Upload File'),
end_form;
}
sub print_results {
my $length;
my $file = param('upload');
if (!$file) {
print "No file uploaded.";
return;
}
print h2('File name'),$file;
print h2('File MIME type'),
uploadInfo($file)->{'Content-Type'};
while (<$file>) {
$length += length($_);
}
print h2('File length'),$length;
}
=head1 Part II: Web Site Care and Feeding
These scripts are designed to make your life as a Webmaster easier,
leaving you time for more exciting things, like tango lessons.
=head2 Logs! Logs! Logs!
Left to their own devices, the log files will grow without limit,
eventually filling up your server's partition and bringing things to a
grinding halt. But wait! Don't turn off logging or throw them away.
Log files are your friends.
=head3 Log rotation
Script II.1.1 shows the basic script for rotating log files. It
renames the current "access_log" to "access_log.0", "access_log.0" to
"access_log.1", and so on. The oldest log gets deleted. Run it from
a cron job to keep your log files from taking over. The faster your
log files grow, the more frequently you should run the script.
Script II.1.1: Basic Log File Rotation
--------------------------------------
#!/usr/local/bin/perl
$LOGPATH='/usr/local/apache/logs';
@LOGNAMES=('access_log','error_log','referer_log','agent_log');
$PIDFILE = 'httpd.pid';
$MAXCYCLE = 4;
chdir $LOGPATH; # Change to the log directory
foreach $filename (@LOGNAMES) {
for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
$oldname = $s ? "$filename.$s" : $filename;
$newname = join(".",$filename,$s+1);
rename $oldname,$newname if -e $oldname;
}
}
kill 'HUP',`cat $PIDFILE`;
=head3 Log rotation and archiving
But some people don't want to delete the old logs. Wow, maybe some
day you could sell them for a lot of money to a marketing and
merchandising company! Script II.1.2 appends the oldest to a gzip
archive. Log files compress extremely well and make great bedtime
reading.
Script II.1.2: Log File Rotation and Archiving
----------------------------------------------
#!/usr/local/bin/perl
$LOGPATH = '/usr/local/apache/logs';
$PIDFILE = 'httpd.pid';
$MAXCYCLE = 4;
$GZIP = '/bin/gzip';
@LOGNAMES=('access_log','error_log','referer_log','agent_log');
%ARCHIVE=('access_log'=>1,'error_log'=>1);
chdir $LOGPATH; # Change to the log directory
foreach $filename (@LOGNAMES) {
system "$GZIP -c $filename.$MAXCYCLE >> $filename.gz"
if -e "$filename.$MAXCYCLE" and $ARCHIVE{$filename};
for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
$oldname = $s ? "$filename.$s" : $filename;
$newname = join(".",$filename,$s+1);
rename $oldname,$newname if -e $oldname;
}
}
kill 'HUP',`cat $PIDFILE`;
=head3 Log rotation, compression and archiving
What's that? Someone broke into your computer, stole your log files
and now B<he's> selling it to a Web marketing and merchandising
company? Shame on them. And on you for letting it happen. Script
II.1.3 uses I<idea> (part of the SSLEay package) to encrypt the log
before compressing it. You need GNU tar to run this one. The log
files are individually compressed and encrypted, and stamped with the
current date.
Script II.1.3: Log File Rotation and Encryption
-----------------------------------------------
#!/usr/local/bin/perl
use POSIX 'strftime';
$LOGPATH = '/home/www/logs';
$PIDFILE = 'httpd.pid';
$MAXCYCLE = 4;
$IDEA = '/usr/local/ssl/bin/idea';
$GZIP = '/bin/gzip';
$TAR = '/bin/tar';
$PASSWDFILE = '/home/www/logs/secret.passwd';
@LOGNAMES=('access_log','error_log','referer_log','agent_log');
%ARCHIVE=('access_log'=>1,'error_log'=>1);
chdir $LOGPATH; # Change to the log directory
foreach $filename (@LOGNAMES) {
my $oldest = "$filename.$MAXCYCLE";
archive($oldest) if -e $oldest and $ARCHIVE{$filename};
for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
$oldname = $s ? "$filename.$s" : $filename;
$newname = join(".",$filename,$s+1);
rename $oldname,$newname if -e $oldname;
}
}
kill 'HUP',`cat $PIDFILE`;
sub archive {
my $f = shift;
my $base = $f;
$base =~ s/\.\d+$//;
my $fn = strftime("$base.%Y-%m-%d_%H:%M.gz.idea",localtime);
system "$GZIP -9 -c $f | $IDEA -kfile $PASSWDFILE > $fn";
system "$TAR rvf $base.tar --remove-files $fn";
}
=head3 Log Parsing
There's a lot you can learn from log files. Script II.1.4 does the
basic access log regular expression match. What you do with the
split-out fields is limited by your imagination. Here's a typical log
entry so that you can follow along (wrapped for readability):
portio.cshl.org - - [03/Feb/1998:17:42:15 -0500]
"GET /pictures/small_logo.gif HTTP/1.0" 200 2172
Script II.1.4: Basic Log Parsing
--------------------------------
#!/usr/local/bin/perl
$REGEX=/^(\S+) (\S+) (\S+) \[([^]]+)\] "(\w+) (\S+).*" (\d+) (\S+)/;
while (<>) {
($host,$rfc931,$user,$date,$request,$URL,$status,$bytes) = m/$REGEX/o;
&collect_some_statistics;
}
&print_some_statistics;
sub collect_some_statistics {
# for you to fill in
}
sub print_some_statistics {
# for you to fill in
}
Script II.1.5 scans the log for certain status codes and prints out the
top URLs or hosts that triggered them. It can be used to get
quick-and-dirty usage statistics, to find broken links, or to detect
certain types of breakin attempts. Use it like this:
% find_status.pl -t10 200 ~www/logs/access_log
TOP 10 URLS/HOSTS WITH STATUS CODE 200:
REQUESTS URL/HOST
-------- --------
1845 /www/wilogo.gif
1597 /cgi-bin/contig/sts_by_name?database=release
1582 /WWW/faqs/www-security-faq.html
1263 /icons/caution.xbm
930 /
886 /ftp/pub/software/WWW/cgi_docs.html
773 /cgi-bin/contig/phys_map
713 /icons/dna.gif
686 /WWW/pics/small_awlogo.gif
Script II.1.5: Find frequent status codes
-----------------------------------------
#!/usr/local/bin/perl
# File: find_status.pl
require "getopts.pl";
&Getopts('L:t:h') || die <<USAGE;
Usage: find_status.pl [-Lth] <code1> <code2> <code3> ...
Scan Web server log files and list a summary
of URLs whose requests had the one of the
indicated status codes.
Options:
-L <domain> Ignore local hosts matching this domain
-t <integer> Print top integer URLS/HOSTS [10]
-h Sort by host rather than URL
USAGE
;
if ($opt_L) {
$opt_L=~s/\./\\./g;
$IGNORE = "(^[^.]+|$opt_L)\$";
}
$TOP=$opt_t || 10;
while (@ARGV) {
last unless $ARGV[0]=~/^\d+$/;
$CODES{shift @ARGV}++;
}
while (<>) {
($host,$rfc931,$user,$date,$request,$URL,$status,$bytes) =
/^(\S+) (\S+) (\S+) \[([^]]+)\] "(\w+) (\S+).*" (\d+) (\S+)/;
next unless $CODES{$status};
next if $IGNORE && $host=~/$IGNORE/io;
$info = $opt_h ? $host : $URL;
$found{$status}->{$info}++;
}
foreach $status (sort {$a<=>$b;} sort keys %CODES) {
$info = $found{$status};
$count = $TOP;
foreach $i (sort {$info->{$b} <=> $info->{$a};} keys %{$info}) {
write;
last unless --$count;
}
$- = 0; # force a new top-of-report
}
format STDOUT_TOP=
TOP @## URLS/HOSTS WITH STATUS CODE @##:
$TOP, $status
REQUESTS URL/HOST
-------- --------
.
format STDOUT=
@##### @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$info->{$i},$i
.
=head3 Offline Reverse DNS Resolution
Many sites turn off reverse name look-ups in order to improve server
performance. The log files will contain the IP addresses of remote
hosts, but not their DNS names. Script II.1.6 will do the reverse name
resolution off-line. You can run it before the log rotation and
archiving scripts, preferably on a machine that isn't busy serving Web
requests at the same time.
This script maintains a cache of resolved names. Because performance
is more important than completeness, if an address doesn't resolve
after two seconds, it moves on to the next one and never tries that
name again.
Script II.1.6: Reverse DNS Resolution
-------------------------------------
#!/usr/local/bin/perl
use constant TIMEOUT => 2;
$SIG{ALRM} = sub {die "timeout"};
while (<>) {
s/^(\S+)/lookup($1)/e;
} continue {
print;
}
sub lookup {
my $ip = shift;
return $ip unless $ip=~/\d+\.\d+\.\d+\.\d+/;
return $CACHE{$ip} if exists $CACHE{$ip};
my @h = eval <<'END';
alarm(TIMEOUT);
my @i = gethostbyaddr(pack('C4',split('\.',$ip)),2);
alarm(0);
@i;
END
$CACHE{$ip} = $h[0];
return $CACHE{$ip} || $ip;
}
=head3 Detecting Robots
I was very upset a few months ago when I did some log analysis and
discovered that 90% of my hits were coming from 10% of users, and that
those 10% were all robots! Script II.1.7 is the script I used to
crunch the log and perform the analysis. The script works like this:
=over
=item 1
we assume that anyone coming from the same IP address with the same
user agent within 30 minutes is the same person/robot (not quite
right, but close enough).
=item 2
anything that fetches /robots.txt is probably a robot, and a "polite"
one, to boot.
=item 3
we count the total number of accesses a user agent makes.
=item 4
we average the interval between successive fetches.
=item 5
we calculate an "index" which is the number of hits over the interval.
Robots have higher indexes than people.
=item 6
we print everything out in a big tab-delimited table for graphing.
=back
By comparing the distribution of "polite" robots to the total
distribution, we can make a good guess as to who the impolite robots
are.
Script II.1.7: Robo-Cop
-----------------------
#!/usr/local/bin/perl
use Time::ParseDate;
use strict 'vars';
# after 30 minutes, we consider this a new session
use constant MAX_INTERVAL => 60*30;
my (%HITS,%INT_NUMERATOR,%INT_DENOMINATOR,%POLITE,%LAST,$HITS);
# This uses a non-standard agent log with lines formatted like this:
# [08/Feb/1998:12:28:35 -0500] phila249-pri.voicenet.com "Mozilla/3.01
(Win95; U)" /cgi-bin/fortune
my $file = shift;
open (IN,$file=~/\.gz$/ ? "zcat $file |" : $file ) || die "Can't open
file/pipe: $!";
while (<IN>) {
my($date,$host,$agent,$URL) = /^\[(.+)\] (\S+) "(.*)" (\S+)$/;
next unless $URL=~/\.(html|htm|txt)$/;
$HITS++;
$host = "$host:$agent"; # concatenate host and agent
$HITS{$host}++;
my $seconds = parsedate($date);
if ($LAST{$host}) {
my $interval = $seconds - $LAST{$host};
if ($interval < MAX_INTERVAL) {
$INT_NUMERATOR{$host} += $interval;
$INT_DENOMINATOR{$host}++;
}
}
$LAST{$host} = $seconds;
$POLITE{$host}++ if $URL eq '/robots.txt';
print STDERR $HITS,"\n" if ($HITS % 1000) == 0;
}
# print out, sorted by hits
print join("\t",qw/Client Robot Hits Interval Hit_Percent Index/),"\n";
foreach (sort {$HITS{$b}<=>$HITS{$a}} keys %HITS) {
next unless $HITS{$_} >= 5; # not enough total hits to mean
much
next unless $INT_DENOMINATOR{$_} >= 5; # not enough consecutive hits
to mean much
my $mean_interval = $INT_NUMERATOR{$_}/$INT_DENOMINATOR{$_};
my $percent_hits = 100*($HITS{$_}/$HITS);
my $index = $percent_hits/$mean_interval;
print join("\t",
$_,
$POLITE{$_} ? 'yes' : 'no',
$HITS{$_},
$mean_interval,
$percent_hits,
$index
),"\n";
}
=head3 Logging to syslog
If you run a large site with many independent servers, you might be
annoyed that they all log into their own file systems rather than into
a central location. Apache offers a little-known feature that allows
it to send its log entries to a process rather than a file. The
process (a Perl script, natch) can do whatever it likes with the logs.
For instance, using Tom Christiansen's C<Syslog> module to send the
info to a remote syslog daemon.
Here's what you add to the Apache httpd.conf file:
<VirtualHost www.company1.com>
CustomLog "| /usr/local/apache/bin/logger company1" common
# blah blah
</VirtualHost>
<VirtualHost www.company2.com>
CustomLog "| /usr/local/apache/bin/logger company2" common
# blah blah
</VirtualHost>
Do the same for each server on the local network.
Here's what you add to each Web server's syslog.conf (this assumes
that the central logging host has the alias hostname "loghost":
local0.info @loghost
Here's what you add to the central log host's syslog.conf:
local0.info /var/log/web/access_log
Script II.1.8 shows the code for the "logger" program:
Script II.1.8 "logger"
----------------------
#!/usr/local/bin/perl
# script: logger
use Sys::Syslog;
$SERVER_NAME = shift || 'www';
$FACILITY = 'local0';
$PRIORITY = 'info';
Sys::Syslog::setlogsock('unix');
openlog ($SERVER_NAME,'ndelay',$FACILITY);
while (<>) {
chomp;
syslog($PRIORITY,$_);
}
closelog;
=head3 Logging to a relational database
One of the selling points of the big commercial Web servers is that
they can log to relational databases via ODBC. Big whoop. With a
little help from Perl, Apache can do that too. Once you've got the
log in a relational database, you can data mine to your heart's
content.
This example uses the freeware mySQL DBMS. To prepare, create an
appropriate database containing a table named "access_log". It should
have a structure like this one. Add whatever indexes you think you
need. Also notice that we truncate URLs at 255 characters. You might
want to use TEXT columns instead.
CREATE TABLE access_log (
when datetime not null,
host varchar(255) not null,
method char(4) not null,
url varchar(255) not null,
auth varchar(50),
browser varchar(50),
referer varchar(255),
status smallint(3) not null,
bytes int(8) default 0
);
Now create the following entries in httpd.conf:
LogFormat "\"%{%Y-%m-%d %H:%M:%S}t\" %h \"%r\" %u \"%{User-agent}i\"
%{Referer}i %s %b" mysql
CustomLog "| /usr/local/apache/bin/mysqllog" mysql
Script II.1.9 is the source code for mysqllog.
Script II.1.9 "mysqllog"
------------------------
#!/usr/local/bin/perl
# script: mysqllog
use DBI;
use constant DSN => 'dbi:mysql:www';
use constant DB_TABLE => 'access_log';
use constant DB_USER => 'nobody';
use constant DB_PASSWD => '';
$PATTERN = '"([^"]+)" (\S+) "(\S+) (\S+) [^"]+" (\S+) "([^"]+)" (\S+) (\d+)
(\S+)';
$db = DBI->connect(DSN,DB_USER,DB_PASSWD) || die DBI->errstr;
$sth = $db->prepare("INSERT INTO ${\DB_TABLE} VALUES(?,?,?,?,?,?,?,?,?)")
|| die $db->errstr;
while (<>) {
chomp;
my($date,$host,$method,$url,$user,$browser,$referer,$status,$bytes) =
/$PATTERN/o;
$user = undef if $user eq '-';
$referer = undef if $referer eq '-';
$browser = undef if $browser eq '-';
$bytes = undef if $bytes eq '-';
$sth->execute($date,$host,$method,$url,$user,$browser,$referer,$status,$bytes);
}
$sth->finish;
$db->disconnect;
NOTE: Your database will grow very quickly. Make sure that you have a
plan for truncating or archiving the oldest entries. Or have a lot of
storage space handy! Also be aware that this will cause a lot of
traffic on your LAN. Better start shopping around for 100BT hubs.
=head2 My server fell down and it can't get up!
Web servers are very stable and will stay up for long periods of time
if you don't mess with them. However, human error can bring them
down, particularly if you have a lot of developers and authors
involved in running the site. The scripts in this section watch the
server and send you an email message when there's a problem.
=head3 Monitoring a local server
The simplest script just tries to signal the Web server process. If
the process has gone away, it sends out an S.O.S. See script II.2.1
shows the technique. Notice that the script has to run as I<root> in
order to successfully signal the server.
Script II.2.1 "localSOS"
------------------------
#!/usr/local/bin/perl
# script: localSOS
use constant PIDFILE => '/usr/local/apache/var/run/httpd.pid';
$MAIL = '/usr/sbin/sendmail';
$MAIL_FLAGS = '-t -oi';
$WEBMASTER = 'webmaster';
open (PID,PIDFILE) || die PIDFILE,": $!\n";
$pid = <PID>; close PID;
kill 0,$pid || sos();
sub sos {
open (MAIL,"| $MAIL $MAIL_FLAGS") || die "mail: $!";
my $date = localtime();
print MAIL <<END;
To: $WEBMASTER
From: The Watchful Web Server Monitor <nobody>
Subject: Web server is down
I tried to call the Web server at $date but there was
no answer.
Respectfully yours,
The Watchful Web Server Monitor
END
close MAIL;
}
=head3 Monitoring a remote server
Local monitoring won't catch problems with remote machines, and
they'll miss subtle problems that can happen when the Web server hangs
but doesn't actually crash. A functional test is better. Script
II.2.2 uses the LWP library to send a HEAD request to a bunch of
servers. If any of them fails to respond, it sends out an SOS. This
script does B<not> have to run as a privileged user.
Script II.2.2 "remoteSOS"
-------------------------
#!/usr/local/bin/perl
# script: remoteSOS
use LWP::Simple;
%SERVERS = (
"Fred's server" => 'http://www.fred.com',
"Martha's server" => 'http://www.stewart-living.com',
"Bill's server" => 'http://www.whitehouse.gov'
);
$MAIL = '/usr/sbin/sendmail';
$MAIL_FLAGS = '-t -oi';
$WEBMASTER = 'webmaster';
foreach (sort keys %SERVERS) {
sos($_) unless head($SERVERS{$_});
}
sub sos {
my $server = shift;
open (MAIL,"| $MAIL $MAIL_FLAGS") || die "mail: $!";
my $date = localtime();
print MAIL <<END;
To: $WEBMASTER
From: The Watchful Web Server Monitor <nobody>
Subject: $server is down
I tried to call $server at $date but there was
no one at home.
Respectfully yours,
The Watchful Web Server Monitor
END
close MAIL;
}
=head3 Resurrecting Dead Servers
So it's not enough to get e-mail that the server's down, you want to
relaunch it as well? Script II.2.3 is a hybrid of localSOS and
remoteSOS that tries to relaunch the local server after sending out
the SOS. It has to be run as B<root>, unless you've made I<apachectl>
suid to root.
Script II.2.2 "webLazarus"
-------------------------
#!/usr/local/bin/perl
# script: webLazarus
use LWP::Simple;
use constant URL => 'http://presto.capricorn.com/';
use constant APACHECTL => '/usr/local/apache/bin/apachectl';
$MAIL = '/usr/sbin/sendmail';
$MAIL_FLAGS = '-t -oi';
$WEBMASTER = '[EMAIL PROTECTED]';
head(URL) || resurrect();
sub resurrect {
open (STDOUT,"| $MAIL $MAIL_FLAGS") || die "mail: $!";
select STDOUT; $| = 1;
open (STDERR,">&STDOUT");
my $date = localtime();
print <<END;
To: $WEBMASTER
From: The Watchful Web Server Monitor <nobody>
Subject: Web server is down
I tried to call the Web server at $date but there was
no answer. I am going to try to resurrect it now:
Mumble, mumble, mumble, shazzzzammmm!
END
;
system APACHECTL,'restart';
print <<END;
That's the best I could do. Hope it helped.
Worshipfully yours,
The Web Monitor
END
close STDERR;
close STDOUT;
}
Here's the message you get when the script is successful:
Date: Sat, 4 Jul 1998 14:55:38 -0400
To: [EMAIL PROTECTED]
Subject: Web server is down
I tried to call the Web server at Sat Jul 4 14:55:37 1998 but there was
no answer. I am going to try to resurrect it now:
Mumble, mumble, mumble, shazzzzammmm!
/usr/local/apache/bin/apachectl restart: httpd not running, trying to start
[Sat Jul 4 14:55:38 1998] [debug] mod_so.c(258): loaded module
setenvif_module
[Sat Jul 4 14:55:38 1998] [debug] mod_so.c(258): loaded module
unique_id_module
/usr/local/apache/bin/apachectl restart: httpd started
That's the best I could do. Hope it helped.
Worshipfully yours,
The Web Monitor
=head2 Site Replication and Mirroring
Often you will want to mirror a page or set of pages from another
server, for example, to distribute the load amongst several replicate
servers, or to keep a set of reference pages handy. The LWP library
makes this easy.
=head3 Mirroring Single Pages
% ./MirrorOne.pl
cats.html: Not Modified
dogs.html: OK
gillie_fish.html: Not Modified
Script II.3.1 mirrorOne.pl
--------------------------
#!/usr/local/bin/perl
# mirrorOne.pl
use LWP::Simple;
use HTTP::Status;
use constant DIRECTORY => '/local/web/price_lists';
%DOCUMENTS = (
'dogs.html' => 'http://www.pets.com/dogs/price_list.html',
'cats.html' => 'http://www.pets.com/cats/price_list.html',
'gillie_fish.html' => 'http://aquaria.com/prices.html'
);
chdir DIRECTORY;
foreach (sort keys %DOCUMENTS) {
my $status = mirror($DOCUMENTS{$_},$_);
warn "$_: ",status_message($status),"\n";
}
=head3 Mirroring a Document Tree
With a little more work, you can recursively mirror an entire set of
linked pages. Script II.3.2 mirrors the requested document and all
subdocuments, using the LWP C<HTML::LinkExtor> module to extract all
the HTML links.
Script II.3.2 mirrorTree.pl
---------------------------
#!/usr/local/bin/perl
# File: mirrorTree.pl
use LWP::UserAgent;
use HTML::LinkExtor;
use URI::URL;
use File::Path;
use File::Basename;
%DONE = ();
my $URL = shift;
$UA = new LWP::UserAgent;
$PARSER = HTML::LinkExtor->new();
$TOP = $UA->request(HTTP::Request->new(HEAD => $URL));
$BASE = $TOP->base;
mirror(URI::URL->new($TOP->request->url));
sub mirror {
my $url = shift;
# get rid of query string "?" and fragments "#"
my $path = $url->path;
my $fixed_url = URI::URL->new ($url->scheme . '://' . $url->netloc .
$path);
# make the URL relative
my $rel = $fixed_url->rel($BASE);
$rel .= 'index.html' if $rel=~m!/$! || length($rel) == 0;
# skip it if we've already done it
return if $DONE{$rel}++;
# create the directory if it doesn't exist already
my $dir = dirname($rel);
mkpath([$dir]) unless -d $dir;
# mirror the document
my $doc = $UA->mirror($fixed_url,$rel);
print STDERR "$rel: ",$doc->message,"\n";
return if $doc->is_error;
# Follow HTML documents
return unless $rel=~/\.html?$/i;
my $base = $doc->base;
# pull out the links and call us recursively
my @links = $PARSER->parse_file("$rel")->links;
my @hrefs = map { url($_->[2],$base)->abs } @links;
foreach (@hrefs) {
next unless is_child($BASE,$_);
mirror($_);
}
}
sub is_child {
my ($base,$url) = @_;
my $rel = $url->rel($base);
return ($rel ne $url) && ($rel !~ m!^[/.]!);
}
=head3 Checking for Bad Links
A slight modification of this last script allows you to check an
entire document hierarchy (your own or someone else's) for bad links.
The script shown in II.3.3 traverses a document, and checks each of the
http:, ftp: and gopher: links to see if there's a response at the
other end. Links that point to sub-documents are fetched and
traversed as before, so you can check your whole site in this way.
% find_bad_links http://prego/apache-1.2/
checking http://prego/apache-1.2/...
checking http://prego/apache-1.2/manual/...
checking http://prego/apache-1.2/manual/misc/footer.html...
checking http://prego/apache-1.2/manual/misc/header.html...
checking http://prego/apache-1.2/manual/misc/nopgp.html...
checking
http://www.yahoo.com/Science/Mathematics/Security_and_Encryption/...
checking http://www.eff.org/pub/EFF/Policy/Crypto/...
checking http://www.quadralay.com/www/Crypt/Crypt.html...
checking http://www.law.indiana.edu/law/iclu.html...
checking http://bong.com/~brian...
checking http://prego/apache-1.2/manual/cgi_path.html...
checking http://www.ics.uci.edu/pub/ietf/http/...
.
.
.
BAD LINKS:
manual/misc/known_bugs.html :
http://www.apache.org/dist/patches/apply_to_1.2b6/
manual/misc/fin_wait_2.html : http://www.freebsd.org/
manual/misc/fin_wait_2.html : http://www.ncr.com/
manual/misc/compat_notes.html : http://www.eit.com/
manual/misc/howto.html : http://www.zyzzyva.com/robots/alert/
manual/misc/perf.html : http://www.software.hp.com/internet/perf/tuning.html
manual/misc/perf.html : http://www.qosina.com/~awm/apache/linux-tcp.html
manual/misc/perf.html :
http://www.sun.com/sun-on-net/Sun.Internet.Solutions/performance/
manual/misc/perf.html : http://www.sun.com/solaris/products/siss/
manual/misc/nopgp.html :
http://www.yahoo.com/Science/Mathematics/Security_and_Encryption/
152 documents checked
11 bad links
Script II.3.2 find_bad_links.pl
-------------------------------
#!/usr/local/bin/perl
# File: find_bad_links.pl
use LWP::UserAgent;
use HTML::LinkExtor;
use URI::URL;
use WWW::RobotRules;
%CAN_HANDLE = ('http'=>1,
'gopher'=>1,
# 'ftp'=>1, # timeout problems?
);
%OUTCOME = ();
$CHECKED = $BAD = 0;
@BAD = ();
my $URL = shift;
$UA = new LWP::UserAgent;
$PARSER = HTML::LinkExtor->new();
$TOP = $UA->request(HTTP::Request->new(HEAD => $URL));
$BASE = $TOP->base;
# handle robot rules
my $robots = URI::URL->new('robots.txt',$BASE->scheme.'://'.$BASE->netloc);
my $robots_text = $UA->request(HTTP::Request->new(GET=>$robots))->content;
$ROBOTRULES = WWW::RobotRules->new;
$ROBOTRULES->parse($robots->abs,$robots_text);
check_links(URI::URL->new($TOP->request->url));
if (@BAD) {
print "\nBAD LINKS:\n";
print join("\n",@BAD),"\n\n";
}
print "$CHECKED documents checked\n",scalar(@BAD)," bad links\n";
sub check_links {
my $url = shift;
my $fixed_url = $url;
$fixed_url =~ s/\#.+$//;
return 1 unless $CAN_HANDLE{$url->scheme};
# check cached outcomes
return $OUTCOME{$fixed_url} if exists $OUTCOME{$fixed_url};
print STDERR "checking $fixed_url...\n";
$CHECKED++;
my $rel = $url->rel($BASE) || 'index.html';
my $child = is_child($BASE,$url);
$UA->timeout(5);
my $doc = $d = $UA->request(HTTP::Request->new(($child ? 'GET' : 'HEAD'
)=>$url));
$OUTCOME{$fixed_url} = $doc->is_success;
return $OUTCOME{$fixed_url}
unless $ROBOTRULES->allowed($fixed_url)
&& $child && $doc->header('Content-type') eq 'text/html';
# Follow HTML documents
my $base = $doc->base;
# pull out the links and call us recursively
my @links = $PARSER->parse($doc->content)->links;
my @hrefs = map { url($_->[2],$base)->abs } @links;
foreach (@hrefs) {
next if check_links($_);
push (@BAD,"$rel : $_");
}
1;
}
sub is_child {
my ($base,$url) = @_;
my $rel = $url->rel($base);
return ($rel ne $url) && ($rel !~ m!^[/.]!);
}
=head2 Load balancing
You've hit the big time, and your site is getting more hits than you
ever dreamed of. Millions, zillions of hits. What's that? System
load just passed 50 and response time is getting kinda' s-l-o-w-w-w?
Perl to the rescue. Set up several replica Web servers with different
hostnames and IP addresses. Run this script on the "main" site and
watch it round-robin the requests to the replica servers. It uses
C<IO::Socket> to listen for incoming requests on port 80. It then
changes its privileges to run as nobody.nogroup, just like a real Web
server. Next it preforks itself a few times (and you always thought
preforking was something fancy, didn't you?), and goes into an
C<accept()> loop. Each time an incoming session comes in, it forks
off another child to handle the request. The child reads the HTTP
request and issues the an HTTP redirection to send the browser to a
randomly selected server.
NOTE: Another way to do this is to have multiple "A" records defined
for your server's hostname and let DNS caching distribute the load.
Script II.4.1: A Load Balancing "Web Server"
--------------------------------------------
#!/usr/local/bin/perl
# list of hosts to balance between
@HOSTS = qw/www1.web.org www2.web.org www3.web.org www4.web.org/;
use IO::Socket;
$SIG{CHLD} = sub { wait() };
$ENV{'PATH'}='/bin:/usr/bin';
chomp($hostname = `/bin/hostname`);
# Listen on port 80
$sock = IO::Socket::INET->new(Listen => 5,
LocalPort => 80,
LocalAddr => $hostname,
Reuse => 1,
Proto => 'tcp');
# become "nobody"
$nobody = (getpwnam('nobody'))[2] || die "nobody is nobody";
$nogroup = (getgrnam('nogroup'))[2] || die "can't grok nogroup";
($<,$() = ($>,$)) = ($nobody,$nogroup); # get rid of root privileges!
($\,$/) = ("\r\n","\r\n\r\n"); # CR/LF on output/input
# Go into server mode
close STDIN; close STDOUT; close STDERR;
# prefork -- gee is that all there is to it?
fork() && fork() && fork() && fork() && exit 0;
# start accepting connections
while (my $s = $sock->accept()) {
do { $s->close; next; } if fork();
my $request = <$s>;
redirect($1,$s) if $request=~/(?:GET|POST|HEAD|PUT)\s+(\S+)/;
$s->flush;
undef $s;
exit 0;
}
sub redirect {
my ($url,$s) = @_;
my $host = $HOSTS[rand(@HOSTS)];
print $s "HTTP/1.0 301 Moved Temporarily";
print $s "Server: Lincoln's Redirector/1.0";
print $s "Location: http://${host}${url}";;
print $s "";
}
=head2 Torture Testing a Server
Any server written in C suffers the risk of static buffer overflow
bugs. In the past, these bugs have led to security compromises and
Web server breakins. Script II.2.3 torture tests servers and CGI
scripts by sending large amounts of random date to them. If the
server crashes, it probably contains a buffer overflow bug.
Here's what you see when a server crashes:
% torture.pl -t 1000 -l 5000 http://www.capricorn.com
torture.pl version 1.0 starting
Base URL: http://www.capricorn.com/cgi-bin/search
Max random data length: 5000
Repetitions: 1000
Post: 0
Append to path: 0
Escape URLs: 0
200 OK
200 OK
200 OK
200 OK
200 OK
500 Internal Server Error
500 Could not connect to www.capricorn.com:80
500 Could not connect to www.capricorn.com:80
500 Could not connect to www.capricorn.com:80
Script II.5.1: torture tester
-----------------------------
#!/usr/local/bin/perl
# file: torture.pl
# Torture test Web servers and scripts by sending them large arbitrary URLs
# and record the outcome.
use LWP::UserAgent;
use URI::Escape 'uri_escape';
require "getopts.pl";
$USAGE = <<USAGE;
Usage: $0 -[options] URL
Torture-test Web servers and CGI scripts
Options:
-l <integer> Max length of random URL to send [1024 bytes]
-t <integer> Number of times to run the test [1]
-P Use POST method rather than GET method
-p Attach random data to path rather than query string
-e Escape the query string before sending it
USAGE
$VERSION = '1.0';
# process command line
&Getopts('l:t:Ppe') || die $USAGE;
# get parameters
$URL = shift || die $USAGE;
$MAXLEN = $opt_l ne '' ? $opt_l : 1024;
$TIMES = $opt_t || 1;
$POST = $opt_P || 0;
$PATH = $opt_p || 0;
$ESCAPE = $opt_e || 0;
# cannot do both a post and a path at the same time
$POST = 0 if $PATH;
# create an LWP agent
my $agent = new LWP::UserAgent;
print <<EOF;
torture.pl version $VERSION starting
Base URL: $URL
Max random data length: $MAXLEN
Repetitions: $TIMES
Post: $POST
Append to path: $PATH
Escape URLs: $ESCAPE
EOF
# Do the test $TIMES times
while ($TIMES) {
# create a string of random stuff
my $garbage = random_string(rand($MAXLEN));
$garbage = uri_escape($garbage) if $ESCAPE;
my $url = $URL;
my $request;
if (length($garbage) == 0) { # if no garbage to add, just fetch URL
$request = new HTTP::Request ('GET',$url);
}
elsif ($POST) { # handle POST request
my $header = new HTTP::Headers (
Content_Type =>
'application/x-www-form-urlencoded',
Content_Length => length($garbage)
);
# garbage becomes the POST content
$request = new HTTP::Request ('POST',$url,$header,$garbage);
} else { # handle GET request
if ($PATH) { # append garbage to the base URL
chop($url) if substr($url,-1,1) eq '/';
$url .= "/$garbage";
} else { # append garbage to the query string
$url .= "?$garbage";
}
$request = new HTTP::Request ('GET',$url);
}
# do the request and fetch the response
my $response = $agent->request($request);
# print the numeric response code and the message
print $response->code,' ',$response->message,"\n";
} continue { $TIMES-- }
# return some random data of the requested length
sub random_string {
my $length = shift;
return undef unless $length >= 1;
return join('',map chr(rand(255)),0..$length-1);
}
For other load testing tools, have a look at our L<Benchmarking
section|guide::performance/Essential_Tools>.
=head1 Part III: mod_perl -- Faster Than a Speeding Bullet
I<mod_perl> is Doug MacEachern's embedded Perl for Apache. With a
mod_perl-enabled server, there's no tedious waiting around while the
Perl interpreter fires up, reads and compiles your script. It's right
there, ready and waiting. What's more, once compiled your script
remains in memory, all charged and raring to go. Suddenly those
sluggish Perl CGI scripts race along at compiled C speeds...or so it
seems.
Most CGI scripts will run unmodified under mod_perl using the
C<Apache::Registry>x CGI compatability layer. But that's not the whole
story. The exciting part is that mod_perl gives you access to the
Apache API, letting you get at the innards of the Apache server and
change its behavior in powerful and interesting ways. This section
will give you a feel for the many things that you can do with
mod_perl.
=head2 Creating Dynamic Pages
This is a ho-hum because you can do it with CGI and with
C<Apache::Registry>. Still, it's worth seeing a simple script written
using the strict mod_perl API so you see what it looks like. Script
III.1.1 prints out a little hello world message.
Install it by adding a section like this one to one of the
configuration files:
<Location /hello/world>
SetHandler perl-script
PerlHandler Apache::Hello
</Location>
Script III.1.1 Apache::Hello
----------------------------
package Apache::Hello;
# file: Apache/Hello.pm
use strict vars;
use Apache::Constants ':common';
sub handler {
my $r = shift;
$r->content_type('text/html');
$r->send_http_header;
my $host = $r->get_remote_host;
$r->print(<<END);
<html>
<head>
<title>Hello There</title>
</head>
<body>
<h1>Hello $host</h1>
Hello to all the nice people at the Perl conference. Lincoln is
trying really hard. Be kind.
</body>
</html>
END
return OK;
}
1;
You can do all the standard CGI stuff, such as reading the query
string, creating fill-out forms, and so on. In fact, C<CGI.pm> works
with mod_perl, giving you the benefit of sticky forms, cookie
handling, and elegant HTML generation.
=head2 File Filters
This is where the going gets fun. With mod_perl, you can install a
I<content handler> that works a lot like a four-letter word
starrer-outer, but a lot faster.
=head3 Adding a Canned Footer to Every Page
Script III.2.1 adds a canned footer to every HTML file. The footer
contains a copyright statement, plus the modification date of the
file. You could easily extend this to add other information, such as
a page hit counter, or the username of the page's owner.
This can be installed as the default handler for all files in a
particular subdirectory like this:
<Location /footer>
SetHandler perl-script
PerlHandler Apache::Footer
</Location>
Or you can declare a new ".footer" extension and arrange for all files
with this extension to be passed through the footer module:
AddType text/html .footer
<Files ~ "\.footer$">
SetHandler perl-script
PerlHandler Apache::Footer
</Files>
Script III.2.1 Apache::Footer
-----------------------------
package Apache::Footer;
# file Apache::Footer.pm
use strict vars;
use Apache::Constants ':common';
use IO::File;
sub handler {
my $r = shift;
return DECLINED unless $r->content_type() eq 'text/html';
my $file = $r->filename;
return DECLINED unless $fh=IO::File->new($file);
my $modtime = localtime((stat($file))[9]);
my $footer=<<END;
<hr>
© 1998 <a href="http://www.ora.com/";>O\'Reilly & Associates</a><br>
<em>Last Modified: $modtime</em>
END
$r->send_http_header;
while (<$fh>) {
s!(</BODY>)!$footer$1!oi;
} continue {
$r->print($_);
}
return OK;
}
1;
For more customized footer/header handling, you might want to look at
the C<Apache::Sandwich> module on CPAN.
=head3 Dynamic Navigation Bar
Sick of hand-coding navigation bars in every HTML page? Less than
enthused by the Java & JavaScript hacks? Here's a dynamic navigation
bar implemented as a server side include.
First create a global configuration file for your site. The first
column is the top of each major section. The second column is the
label to print in the navigation bar
# Configuration file for the navigation bar
/index.html Home
/new/ What's New
/tech/ Tech Support
/download/ Download
/dev/zero Customer support
/dev/null Complaints
Then, at the top (or bottom) of each HTML page that you want the
navigation bar to appear on, add this comment:
<!--#NAVBAR-->
Now add C<Apache::NavBar> to your system (Script III.2.2). This module
parses the configuration file to create a "navigation bar object". We
then call the navigation bar object's C<to_html()> method in order to
generate the HTML for the navigation bar to display on the current
page (it will be different for each page, depending on what major
section the page is in).
The next section does some checking to avoid transmitting the page
again if it is already cached on the browser. The effective last
modified time for the page is either the modification time of its HTML
source code, or the navbar's configuration file modification date,
whichever is more recent.
The remainder is just looping through the file a section at a time,
searching for the C<E<lt>!--NAVBAR--E<gt>> comment, and substituting
the navigation bar HTML.
Script III.2.2 Apache::NavBar
-----------------------------
package Apache::NavBar;
# file Apache/NavBar.pm
use strict;
use Apache::Constants qw(:common);
use Apache::File ();
my %BARS = ();
my $TABLEATTS = 'WIDTH="100%" BORDER=1';
my $TABLECOLOR = '#C8FFFF';
my $ACTIVECOLOR = '#FF0000';
sub handler {
my $r = shift;
my $bar = read_configuration($r) || return DECLINED;
$r->content_type eq 'text/html' || return DECLINED;
my $fh = Apache::File->new($r->filename) || return DECLINED;
my $navbar = $bar->to_html($r->uri);
$r->update_mtime($bar->modified);
$r->set_last_modified;
my $rc = $r->meets_conditions;
return $rc unless $rc == OK;
$r->send_http_header;
return OK if $r->header_only;
local $/ = "";
while (<$fh>) {
s:<!--NAVBAR-->:$navbar:oi;
} continue {
$r->print($_);
}
return OK;
}
# read the navigation bar configuration file and return it as a
# hash.
sub read_configuration {
my $r = shift;
my $conf_file;
return unless $conf_file = $r->dir_config('NavConf');
return unless -e ($conf_file = $r->server_root_relative($conf_file));
my $mod_time = (stat _)[9];
return $BARS{$conf_file} if $BARS{$conf_file}
&& $BARS{$conf_file}->modified >= $mod_time;
return $BARS{$conf_file} = NavBar->new($conf_file);
}
package NavBar;
# create a new NavBar object
sub new {
my ($class,$conf_file) = @_;
my (@c,%c);
my $fh = Apache::File->new($conf_file) || return;
while (<$fh>) {
chomp;
s/^\s+//; s/\s+$//; #fold leading and trailing whitespace
next if /^#/ || /^$/; # skip comments and empty lines
next unless my($url, $label) = /^(\S+)\s+(.+)/;
push @c, $url; # keep the url in an ordered array
$c{$url} = $label; # keep its label in a hash
}
return bless {'urls' => [EMAIL PROTECTED],
'labels' => \%c,
'modified' => (stat $conf_file)[9]}, $class;
}
# return ordered list of all the URIs in the navigation bar
sub urls { return @{shift->{'urls'}}; }
# return the label for a particular URI in the navigation bar
sub label { return $_[0]->{'labels'}->{$_[1]} || $_[1]; }
# return the modification date of the configuration file
sub modified { return $_[0]->{'modified'}; }
sub to_html {
my $self = shift;
my $current_url = shift;
my @cells;
for my $url ($self->urls) {
my $label = $self->label($url);
my $is_current = $current_url =~ /^$url/;
my $cell = $is_current ?
qq(<FONT COLOR="$ACTIVECOLOR">$label</FONT>)
: qq(<A HREF="$url">$label</A>);
push @cells,
qq(<TD CLASS="navbar" ALIGN=CENTER
BGCOLOR="$TABLECOLOR">$cell</TD>\n);
}
return qq(<TABLE $TABLEATTS><TR>@cells</TR></TABLE>\n);
}
1;
__END__
<Location />
SetHandler perl-script
PerlHandler Apache::NavBar
PerlSetVar NavConf etc/navigation.conf
</Location>
C<Apache::NavBar> is available on the CPAN, with further improvements.
=head3 On-the-Fly Compression
WU-FTP has a great feature that automatically gzips a file if you
fetch it by name with a I<.gz> extension added. Why can't Web servers
do that trick? With Apache and mod_perl, you can.
Script III.2.4 is a content filter that automatically gzips everything
retrieved from a particular directory and adds the "gzip"
C<Content-Encoding> header to it. Unix versions of Netscape Navigator
will automatically recognize this encoding type and decompress the
file on the fly. Windows and Mac versions don't. You'll have to save
to disk and decompress, or install the WinZip plug-in. Bummer.
The code uses the C<Compress::Zlib> module, and has to do a little
fancy footwork (but not too much) to create the correct gzip header.
You can extend this idea to do on-the-fly encryption, or whatever you
like.
Here's the configuration entry you'll need. Everything in the
I</compressed> directory will be compressed automagically.
<Location /compressed>
SetHandler perl-script
PerlHandler Apache::GZip
</Location>
Script III.2.3: Apache::GZip
----------------------------
package Apache::GZip;
#File: Apache::GZip.pm
use strict vars;
use Apache::Constants ':common';
use Compress::Zlib;
use IO::File;
use constant GZIP_MAGIC => 0x1f8b;
use constant OS_MAGIC => 0x03;
sub handler {
my $r = shift;
my ($fh,$gz);
my $file = $r->filename;
return DECLINED unless $fh=IO::File->new($file);
$r->header_out('Content-Encoding'=>'gzip');
$r->send_http_header;
return OK if $r->header_only;
tie *STDOUT,'Apache::GZip',$r;
print($_) while <$fh>;
untie *STDOUT;
return OK;
}
sub TIEHANDLE {
my($class,$r) = @_;
# initialize a deflation stream
my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef;
# gzip header -- don't ask how I found out
$r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC));
return bless { r => $r,
crc => crc32(undef),
d => $d,
l => 0
},$class;
}
sub PRINT {
my $self = shift;
foreach (@_) {
# deflate the data
my $data = $self->{d}->deflate($_);
$self->{r}->print($data);
# keep track of its length and crc
$self->{l} += length($_);
$self->{crc} = crc32($_,$self->{crc});
}
}
sub DESTROY {
my $self = shift;
# flush the output buffers
my $data = $self->{d}->flush;
$self->{r}->print($data);
# print the CRC and the total length (uncompressed)
$self->{r}->print(pack("LL",@{$self}{qw/crc l/}));
}
1;
For some alternatives that are being maintained, you might want to
look at the C<Apache::Compress> and C<Apache::GzipChain> modules on
CPAN, which can handle the output of any handler in a chain.
By adding a URI translation handler, you can set things up so that a
remote user can append a I<.gz> to the end of any URL and the file we
be delivered in compressed form. Script III.2.4 shows the translation
handler you need. It is called during the initial phases of the
request to make any modifications to the URL that it wishes. In this
case, it removes the I<.gz> ending from the filename and arranges for
C<Apache:GZip> to be called as the content handler. The
C<lookup_uri()> call is used to exclude anything that has a special
handler already defined (such as CGI scripts), and actual gzip files.
The module replaces the information in the request object with
information about the real file (without the C<.gz>), and arranges for
C<Apache::GZip> to be the content handler for this file.
You just need this one directive to activate handling for all URLs at
your site:
PerlTransHandler Apache::AutoGZip
Script III.2.4: Apache::AutoGZip
--------------------------------
package Apache::AutoGZip;
use strict 'vars';
use Apache::Constants qw/:common/;
sub handler {
my $r = shift;
# don't allow ourselves to be called recursively
return DECLINED unless $r->is_initial_req;
# don't do anything for files not ending with .gz
my $uri = $r->uri;
return DECLINED unless $uri=~/\.gz$/;
my $basename = $`;
# don't do anything special if the file actually exists
return DECLINED if -e $r->lookup_uri($uri)->filename;
# look up information about the file
my $subr = $r->lookup_uri($basename);
$r->uri($basename);
$r->path_info($subr->path_info);
$r->filename($subr->filename);
# fix the handler to point to Apache::GZip;
my $handler = $subr->handler;
unless ($handler) {
$r->handler('perl-script');
$r->push_handlers('PerlHandler','Apache::GZip');
} else {
$r->handler($handler);
}
return OK;
}
1;
=head2 Access Control
Access control, as opposed to authentication and authorization, is
based on something the user "is" rather than something he "knows".
The "is" is usually something about his browser, such as its IP
address, hostname, or user agent. Script III.3.1 blocks access to the
Web server for certain User Agents (you might use this to block
impolite robots).
C<Apache::BlockAgent> reads its blocking information from a "bad
agents" file, which contains a series of pattern matches. Most of the
complexity of the code comes from watching this file and recompiling
it when it changes. If the file doesn't change, it's is only read
once and its patterns compiled in memory, making this module fast.
Here's an example bad agents file:
^teleport pro\/1\.28
^nicerspro
^mozilla\/3\.0 \(http engine\)
^netattache
^crescent internet toolpak http ole control v\.1\.0
^go-ahead-got-it
^wget
^devsoft's http component v1\.0
^www\.pl
^digout4uagent
A configuration entry to activate this blocker looks like this. In
this case we're blocking access to the entire site. You could also
block access to a portion of the site, or have different bad agents
files associated with different portions of the document tree.
<Location />
PerlAccessHandler Apache::BlockAgent
PerlSetVar BlockAgentFile /home/www/conf/bad_agents.txt
</Location>
Script III.3.1: Apache::BlockAgent
----------------------------------
package Apache::BlockAgent;
# block browsers that we don't like
use strict 'vars';
use Apache::Constants ':common';
use IO::File;
my %MATCH_CACHE;
my $DEBUG = 0;
sub handler {
my $r = shift;
return DECLINED unless my $patfile = $r->dir_config('BlockAgentFile');
return FORBIDDEN unless my $agent = $r->header_in('User-Agent');
return SERVER_ERROR unless my $sub = get_match_sub($r,$patfile);
return OK if $sub->($agent);
$r->log_reason("Access forbidden to agent $agent",$r->filename);
return FORBIDDEN;
}
# This routine creates a pattern matching subroutine from a
# list of pattern matches stored in a file.
sub get_match_sub {
my ($r,$filename) = @_;
my $mtime = -M $filename;
# try to return the sub from cache
return $MATCH_CACHE{$filename}->{'sub'} if
$MATCH_CACHE{$filename} &&
$MATCH_CACHE{$filename}->{'mod'} <= $mtime;
# if we get here, then we need to create the sub
return undef unless my $fh = new IO::File($filename);
chomp(my @pats = <$fh>); # get the patterns into an array
my $code = "sub { \$_ = shift;\n";
foreach (@pats) {
next if /^#/
$code .= "return undef if /$_/i;\n";
}
$code .= "1; }\n";
warn $code if $DEBUG;
# create the sub, cache and return it
my $sub = eval $code;
unless ($sub) {
$r->log_error($r->uri,": ",$@);
return undef;
}
@{$MATCH_CACHE{$filename}}{'sub','mod'}=($sub,$modtime);
return $MATCH_CACHE{$filename}->{'sub'};
}
1;
=head2 Authentication and Authorization
Thought you were stuck with authentication using text, DBI and DBM
files? mod_perl opens the authentication/authorization API wide. The
two phases are authentication, in which the user has to prove who he
or she is (usually by providing a username and password), and
authorization, in which the system decides whether this user has
sufficient privileges to view the requested URL. A scheme can
incorporate authentication and authorization either together or
singly.
=head3 Authentication with NIS
If you keep Unix system passwords in I</etc/passwd> or distribute them
by NIS (not NIS+) you can authenticate Web users against the system
password database. (It's not a good idea to do this if the system is
connected to the Internet because passwords travel in the clear, but
it's OK for trusted intranets.)
Script III.4.1 shows how the C<Apache::AuthSystem> module fetches the
user's name and password, compares it to the system password, and
takes appropriate action. The C<getpwnam()> function operates either
on local files or on the NIS database, depending on how the server
host is configured. WARNING: the module will fail if you use a shadow
password system, since the Web server doesn't have root privileges.
In order to activate this system, put a configuration directive like
this one in access.conf:
<Location /protected>
AuthName Test
AuthType Basic
PerlAuthenHandler Apache::AuthSystem
require valid-user
</Location>
Script III.4.1: Apache::AuthSystem
----------------------------------
package Apache::AuthSystem;
# authenticate users on system password database
use strict;
use Apache::Constants ':common';
sub handler {
my $r = shift;
my($res, $sent_pwd) = $r->get_basic_auth_pw;
return $res if $res != OK;
my $user = $r->connection->user;
my $reason = "";
my($name,$passwd) = getpwnam($user);
if (!$name) {
$reason = "user does not have an account on this system";
} else {
$reason = "user did not provide correct password"
unless $passwd eq crypt($sent_pwd,$passwd);
}
if($reason) {
$r->note_basic_auth_failure;
$r->log_reason($reason,$r->filename);
return AUTH_REQUIRED;
}
return OK;
}
1;
There are modules doing equivalent things on CPAN:
C<Apache::AuthenPasswd> and C<Apache::AuthxPasswd>.
=head3 Anonymous Authentication
Here's a system that authenticates users the way anonymous FTP does.
They have to enter a name like "Anonymous" (configurable) and a
password that looks like a valid e-mail address. The system rejects
the username and password unless they are formatted correctly.
In a real application, you'd probably want to log the password
somewhere for posterity. Script III.4.2 shows the code for
C<Apache::AuthAnon>. To activate it, create a I<httpd.conf> section
like this one:
<Location /protected>
AuthName Anonymous
AuthType Basic
PerlAuthenHandler Apache::AuthAnon
require valid-user
PerlSetVar Anonymous anonymous|anybody
</Location>
Script III.4.2: Anonymous Authentication
----------------------------------------
package Apache::AuthAnon;
use strict;
use Apache::Constants ':common';
my $email_pat = '[EMAIL PROTECTED]';
my $anon_id = "anonymous";
sub handler {
my $r = shift;
my($res, $sent_pwd) = $r->get_basic_auth_pw;
return $res if $res != OK;
my $user = lc $r->connection->user;
my $reason = "";
my $check_id = $r->dir_config("Anonymous") || $anon_id;
unless($user =~ /^$check_id$/i) {
$reason = "user did not enter a valid anonymous username";
}
unless($sent_pwd =~ /$email_pat/o) {
$reason = "user did not enter an email address password";
}
if($reason) {
$r->note_basic_auth_failure;
$r->log_reason($reason,$r->filename);
return AUTH_REQUIRED;
}
$r->notes(AuthAnonPassword => $sent_pwd);
return OK;
}
1;
=head3 Gender-Based Authorization
After authenticating, you can authorize. The most familiar type of
authorization checks a group database to see if the user belongs to
one or more privileged groups. But authorization can be anything you
dream up.
Script III.4.3 shows how you can authorize users by their gender (or at
least their I<apparent> gender, by checking their names with Jon
Orwant's C<Text::GenderFromName> module. This must be used in
conjunction with an authentication module, such as one of the standard
Apache modules or a custom one.
This configuration restricts access to users with feminine names,
except for the users "Webmaster" and "Jeff", who are allowed access.
<Location /ladies_only>
AuthName "Ladies Only"
AuthType Basic
AuthUserFile /home/www/conf/users.passwd
PerlAuthzHandler Apache::AuthzGender
require gender F # allow females
require user Webmaster Jeff # allow Webmaster or Jeff
</Location>
The script uses a custom error response to explain why the user was
denied admittance. This is better than the standard "Authorization
Failed" message.
Script III.4.3: Apache::AuthzGender
-----------------------------------
package Apache::AuthzGender;
use strict;
use Text::GenderFromName;
use Apache::Constants ":common";
my %G=('M'=>"male",'F'=>"female");
sub handler {
my $r = shift;
return DECLINED unless my $requires = $r->requires;
my $user = lc($r->connection->user);
substr($user,0,1)=~tr/a-z/A-Z/;
my $guessed_gender = uc(gender($user)) || 'M';
my $explanation = <<END;
<html><head><title>Unauthorized</title></head><body>
<h1>You Are Not Authorized to Access This Page</h1>
Access to this page is limited to:
<ol>
END
foreach (@$requires) {
my ($requirement,@rest ) = split(/\s+/,$_->{requirement});
if (lc $requirement eq 'user') {
foreach (@rest) { return OK if $user eq $_; }
$explanation .= "<LI>Users @rest.\n";
} elsif (lc $requirement eq 'gender') {
foreach (@rest) { return OK if $guessed_gender eq uc $_; }
$explanation .= "<LI>People of the @[EMAIL PROTECTED]
persuasion.\n";
} elsif (lc $requirement eq 'valid-user') {
return OK;
}
}
$explanation .= "</OL></BODY></HTML>";
$r->custom_response(AUTH_REQUIRED,$explanation);
$r->note_basic_auth_failure;
$r->log_reason("user $user: not authorized",$r->filename);
return AUTH_REQUIRED;
}
1;
C<Apache::AuthzGender> is available from the CPAN.
=head2 Proxy Services
mod_perl gives you access to Apache's ability to act as a Web proxy.
You can intervene at any step in the proxy transaction to modify the
outgoing request (for example, stripping off headers in order to
create an anonymizing proxy) or to modify the returned page.
=head3 A Banner Ad Blocker
Script III.5.1 shows the code for a banner-ad blocker written by Doug
MacEachern. It intercepts all proxy requests, substituting its own
content handler for the default. The content handler uses the LWP
library to fetch the requested document. If the retrieved document is
an image, and its URL matches the pattern (ads?|advertisement|banner),
then the content of the image is replaced with a dynamically-generated
GIF that reads "Blocked Ad". The generated image is exactly the same
size as the original, preserving the page layout. Notice how the
outgoing headers from the Apache request object are copied to the LWP
request, and how the incoming LWP response headers are copied back to
Apache. This makes the transaction nearly transparent to Apache and
to the remote server.
In addition to LWP you'll need C<GD.pm> and C<Image::Size> to run this
module. To activate it, add the following line to the configuration
file:
PerlTransHandler Apache::AdBlocker
Then configure your browser to use the server to proxy all its HTTP
requests. Works like a charm! With a little more work, and some help
from the C<ImageMagick> module, you could adapt this module to
quiet-down animated GIFs by stripping them of all but the very first
frame.
Script III.5.1: Apache::AdBlocker
---------------------------------
package Apache::AdBlocker;
use strict;
use vars qw(@ISA $VERSION);
use Apache::Constants qw(:common);
use GD ();
use Image::Size qw(imgsize);
use LWP::UserAgent ();
@ISA = qw(LWP::UserAgent);
$VERSION = '1.00';
my $UA = __PACKAGE__->new;
$UA->agent(join "/", __PACKAGE__, $VERSION);
my $Ad = join "|", qw{ads? advertisement banner};
sub handler {
my($r) = @_;
return DECLINED unless $r->proxyreq;
$r->handler("perl-script"); #ok, let's do it
$r->push_handlers(PerlHandler => \&proxy_handler);
return OK;
}
sub proxy_handler {
my($r) = @_;
my $request = HTTP::Request->new($r->method, $r->uri);
$r->headers_in->do(sub {
$request->header(@_);
});
# copy POST data, if any
if($r->method eq 'POST') {
my $len = $r->header_in('Content-length');
my $buf;
$r->read($buf, $len);
$request->content($buf);
$request->content_type($r->content_type);
}
my $response = $UA->request($request);
$r->content_type($response->header('Content-type'));
#feed response back into our request_rec*
$r->status($response->code);
$r->status_line(join " ", $response->code, $response->message);
$response->scan(sub {
$r->header_out(@_);
});
if ($r->header_only) {
$r->send_http_header();
return OK;
}
my $content = \$response->content;
if($r->content_type =~ /^image/ and $r->uri =~ /\b($Ad)\b/i) {
block_ad($content);
$r->content_type("image/gif");
}
$r->content_type('text/html') unless $$content;
$r->send_http_header;
$r->print($$content || $response->error_as_HTML);
return OK;
}
sub block_ad {
my $data = shift;
my($x, $y) = imgsize($data);
my $im = GD::Image->new($x,$y);
my $white = $im->colorAllocate(255,255,255);
my $black = $im->colorAllocate(0,0,0);
my $red = $im->colorAllocate(255,0,0);
$im->transparent($white);
$im->string(GD::gdLargeFont(),5,5,"Blocked Ad",$red);
$im->rectangle(0,0,$x-1,$y-1,$black);
$$data = $im->gif;
}
1;
Another way of doing this module would be to scan all proxied HTML
files for C<E<lt>imgE<gt>> tags containing one of the verboten URLs,
then replacing the C<src> attribute with a transparent GIF of our own.
However, unless the C<E<lt>imgE<gt>> tag contained C<width> and
C<height> attributes, we wouldn't be able to return a GIF of the
correct size -- unless we were to go hunting for the GIF with LWP, in
which case we might as well do it this way.
=head2 Customized Logging
After Apache handles a transaction, it passes all the information
about the transaction to the log handler. The default log handler
writes out lines to the log file. With mod_perl, you can install your
own log handler to do customized logging.
=head3 Send E-Mail When a Particular Page Gets Hit
Script III.6.1 installs a log handler which watches over a page or set
of pages. When someone fetches a watched page, the log handler sends
off an e-mail to notify someone (probably the owner of the page) that
the page has been read.
To activate the module, just attach a C<PerlLogHandler> to the
C<E<lt>LocationE<gt>> or C<E<lt>FilesE<gt>> you wish to watch. For
example:
<Location /~lstein>
PerlLogHandler Apache::LogMail
PerlSetVar mailto [EMAIL PROTECTED]
</Location>
The "mailto" directive specifies the name of the recipient(s) to
notify.
Script III.6.1: Apache::LogMail
-------------------------------
package Apache::LogMail;
use Apache::Constants ':common';
sub handler {
my $r = shift;
my $mailto = $r->dir_config('mailto');
return DECLINED unless $mailto
my $request = $r->the_request;
my $uri = $r->uri;
my $agent = $r->header_in("User-agent");
my $bytes = $r->bytes_sent;
my $remote = $r->get_remote_host;
my $status = $r->status_line;
my $date = localtime;
unless (open (MAIL,"|/usr/lib/sendmail -oi -t")) {
$r->log_error("Couldn't open mail: $!");
return DECLINED;
}
print MAIL <<END;
To: $mailto
From: Mod Perl <webmaster>
Subject: Somebody looked at $uri
At $date, a user at $remote looked at
$uri using the $agent browser.
The request was $request,
which resulted returned a code of $status.
$bytes bytes were transferred.
END
close MAIL;
return OK;
}
1;
=head3 Writing Log Information Into a Relational Database
Coming full circle, Script III.6.2 shows a module that writes log
information into a DBI database. The idea is similar to Script I.1.9,
but there's now no need to open a pipe to an external process. It's
also a little more efficient, because the log data fields can be
recovered directly from the Apache request object, rather than parsed
out of a line of text. Another improvement is that we can set up the
Apache configuration files so that only accesses to certain
directories are logged in this way.
To activate, add something like this to your configuration file:
PerlLogHandler Apache::LogDBI
Or, to restrict special logging to accesses of files in below the URL
"/lincoln_logs" add this:
<Location /lincoln_logs>
PerlLogHandler Apache::LogDBI
</Location>
Script III.6.2: Apache::LogDBI
------------------------------
package Apache::LogDBI;
use Apache::Constants ':common';
use strict 'vars';
use vars qw($DB $STH);
use DBI;
use POSIX 'strftime';
use constant DSN => 'dbi:mysql:www';
use constant DB_TABLE => 'access_log';
use constant DB_USER => 'nobody';
use constant DB_PASSWD => '';
$DB = DBI->connect(DSN,DB_USER,DB_PASSWD) || die DBI->errstr;
$STH = $DB->prepare("INSERT INTO ${\DB_TABLE} VALUES(?,?,?,?,?,?,?,?,?)")
|| die $DB->errstr;
sub handler {
my $r = shift;
my $date = strftime('%Y-%m-%d %H:%M:%S',localtime);
my $host = $r->get_remote_host;
my $method = $r->method;
my $url = $r->uri;
my $user = $r->connection->user;
my $referer = $r->header_in('Referer');
my $browser = $r->header_in("User-agent");
my $status = $r->status;
my $bytes = $r->bytes_sent;
$STH->execute($date,$host,$method,$url,$user,
$browser,$referer,$status,$bytes);
return OK;
}
1;
There are other alternatives which are more actively maintained
available from the CPAN: C<Apache::DBILogger> and
C<Apache::DBILogConfig>.
=head1 Conclusion
These tricks illustrate the true power of mod_perl; not only were Perl
and Apache good friends from the start, thanks to Perl's excellent
text-handling capacity, but when mod_perl is used, your complete
access to the Apache API gives you unprecendented power in dynamic web
serving.
To find more tips and tricks, look for modules on the CPAN, look
through the mod_perl L<documentation|docs::index>, and also in the
following books by Lincoln Stein:
=over 4
=item "How to Set Up and Maintain a Web Site"
General introduction to Web site care and feeding, with an emphasis on
Apache. Addison-Wesley 1997.
Companion Web site at http://www.genome.wi.mit.edu/WWW/
=item "Web Security, a Step-by-Step Reference Guide"
How to keep your Web site free from thieves, vandals, hooligans and
other yahoos. Addison-Wesley 1998.
Companion Web site at http://www.w3.org/Security/Faq/
=item "The Official Guide to Programming with CGI.pm"
Everything I know about CGI.pm (and some things I don't!). John Wiley
& Sons, 1998.
Companion Web site at http://www.wiley.com/compbooks/stein/
=item "Writing Apache Modules in Perl and C"
Co-authored with Doug MacEachern. O'Reilly & Associates.
Companion Web site at http://www.modperl.com/
=item WebTechniques Columns
I write a monthly column for WebTechniques magazine. You can find
back-issues and reprints at http://www.web-techniques.com/
=item The Perl Journal Columns
I write a quarterly column for TPJ. Source code listings are available
at http://www.tpj.com/
=back
=head1 Maintainers
The maintainer is the person(s) you should contact with updates,
corrections and patches.
=over
=item *
Per Einar Ellefsen E<lt>per.einar (at) skynet.beE<gt>
=back
=head1 Authors
=over
=item * Lincoln Stein E<lt>lstein (at) cshl.orgE<gt>
=back
Only the major authors are listed above. For contributors see the
Changes file.
=cut
1.1
modperl-docs/src/docs/tutorials/tmpl/comparison/comparison.pod
Index: comparison.pod
===================================================================
=head1 NAME
Choosing a Templating System
=head1 Description
Everything you wanted to know about templating systems and didn't dare
to ask. Well, not everything....
=head1 Introduction
Go on, admit it: you've written a templating system. It's okay,
nearly everyone has at some point. You start out with something
beautifully simple like C<$HTML =~ s/\$(\w+)/${$1}/g> and end up
adding conditionals and loops and includes until you've created your
very own unmaintainable monster.
Luckily for you, you are not the first to think it might be nice to get
the HTML out of your code. Many have come before, and more than a few
have put their contributions up on CPAN. At this time, there are so
many templating modules on CPAN that it's almost certain you can find
one that meets your needs. This document aims to be your guide to
those modules, leading you down the path to the templating system of
your dreams.
And, if you just went straight to CPAN in the first place and never
bothered to write your own, congratulations: you're one step ahead of
the rest of us.
=head2 On A Personal Note
Nothing can start an argument faster on the mod_perl mailing list than
a claim that one approach to templating is better than another.
People get very attached to the tools they've chosen. Therefore, let
me say up front that I am biased. I've been at this for a while and I
have opinions about what works best. I've tried to present a balanced
appraisal of the features of various systems in this document, but it
probably won't take you long to figure out what I like. Besides,
attempts to be completely unbiased lead to useless documents that
don't contain any real information. So take it all with a pound of
salt and if you think I've been unfair to a particular tool through a
factual error or omission, let me know.
=head1 Why Use Templates?
Why bother using templates at all? Print statements and CGI.pm were
good enough for Grandpa, so why should you bother learning a new way
to do things?
=head2 Consistency of Appearance
It doesn't take a genius to see that making one navigation bar
template and using it in all of your pages is easier to manage than
hard-coding it every where. If you build your whole site like this,
it's much easier to make site-wide changes in the look and feel.
=head2 Reusability
Along the same lines, building a set of commonly used components makes
it easier to create new pages.
=head2 Better Isolation from Changes
Which one changes more often, the logic of your application or the
HTML used to display it? It actually doesn't matter which you
answered, as long as it's one of them. Templates can be a great
abstraction layer between the application logic and the display logic,
allowing one to be updated without touching the other.
=head2 Division of Labor
Separating your Perl code from your HTML means that when your
marketing department decides everything should be green instead of
blue, you don't have to lift a finger. Just send them to the HTML
coder down the hall. It's a beautiful thing, getting out of the HTML
business.
Even if the same people in your organization write the Perl code and
the HTML, you at last have the opportunity for more people to be
working on the project in parallel.
=head1 What Are the Differences?
Before we look at the available options, let's go through an
explanation of some of the things that make them different.
=head2 Execution Models
Although some try to be flexible about it, most templating systems
expect you to use some variation of the two basic execution models,
which I will refer to as "pipeline" and "callback." In the callback
style, you let the template take over and it has the application's
control flow coded into it. It uses callbacks to modules or snippets
of in-line Perl code to retrieve data for display or perform actions
like user authentication. Some popular examples of systems using this
model include Mason, Embperl, and Apache::ASP.
The pipeline style does all the work up front in a standard CGI or
mod_perl handler, then decides which template to run and passes some
data to it. The template has no control flow logic in it, just
presentation logic, e.g. show this graphic if this item is on sale.
Popular systems supporting this approach include HTML::Template and
Template Toolkit.
The callback model works very well for publishing-oriented sites where
the pages are essentially mix and match sets of articles and lists.
Ideally, a site can be broken down into visual "components" or pieces
of pages which are general enough for an HTML coder to re-combine them
into entirely new kinds of pages without any help from a programmer.
The callback model can get a bit hairy when you have to code logic
that can result in totally different content being returned. For
example, if you have a system that processes some form input and takes
the user to different pages depending on the data submitted. In these
situations, it's easy to end up coding a spaghetti of includes and
redirects, or putting what are really multiple pages in the same file.
On the other hand, a callback approach can result in fewer files (if
the Perl code is in the HTML file), and feels easier and more
intuitive to many developers. It's a simple step from static files to
static files with a few in-line snippets of code in them. This is part
of why PHP is so popular with new developers.
The pipeline model is more like a traditional model-view-controller
design. Working this way can provide additional performance tuning
opportunities over an approach where you don't know what data will be
needed at the beginning of the request. You can aggregate database
queries, make smarter choices about caching, etc. It can also promote
a cleaner separation of application logic and presentation. However,
this approach takes longer to get started with since it's a bigger
conceptual hurdle and always involves at least two files: one for the
Perl code and one for the template.
Keep in mind, many systems offer significant flexibility for
customizing their execution models. For example, Mason users could
write separate components for application logic and display, letting
the logic components choose which display component to run after
fetching their data. This allows it to be used in a pipeline style.
A Template Toolkit application could be written to use a simple
generic handler (like the Apache::Template module included in the
distribution) with all the application logic placed in the template
using object calls or in-line Perl. This would be using it in a
callback style.
HTML::Template and some of the AxKit XML processors are fairly rigid
about insisting on a pipeline approach. Neither provide methods for
calling back into Perl code during the HTML formatting stage; you have
to do the work before running the template. The authors of these
tools consider this a feature since it prevents developers from
cheating on the separation of application code and presentation.
=head2 Languages
Here's the big issue with templating systems. This is the one that
always cranks up the flame on web development mailing lists.
Some systems use in-line Perl statements. They may provide some extra
semantics, like Embperl's operators for specifying whether the code's
output should be displayed or Mason's C<E<lt>%initE<gt>> sections for
specifying
when the code gets run, but at the end of the day your templates are
written in Perl.
Other systems provide a specialized mini-language instead of (or in
addition to) in-line Perl. These will typically have just enough
syntax to handle variable substitution, conditionals, and looping.
HTML::Template and Template Toolkit are popular systems using this
approach. AxKit straddles the fence, providing both a (not-so-)
mini-language - XSLT - and an in-line Perl approach - XPathScript.
Here's how a typical discussion of the merits of these approaches might go:
B<IN-LINE:> Mini-languages are stupid. I already know Perl and it's easy
enough. Why would you want to use something different?
B<MINI-LANG:> Because my HTML coder doesn't know Perl, and this is easier
for him.
B<IN-LINE:> Maybe he should learn some Perl. He'd get paid more.
B<MINI-LANG:> Whatever. You just want to use in-line Perl so you can
handle change requests by putting little hacks in the template instead
of changing your modules. That's sloppy coding.
B<IN-LINE:> That's efficient coding. I can knock out data editing
screens in half the time it takes you, and then I can go back through,
putting all the in-line code into modules and just have the templates
call them.
B<MINI-LANG:> You could, but you won't.
B<IN-LINE:> Is it chilly up there in that ivory tower?
B<MINI-LANG:> Go write some VBScript, weenie.
etc.
Most people pick a side in this war and stay there. If you are one of
the few who hasn't fully decided yet, you should take a moment to
think about who will be building and maintaining your templates, what
skills those people have, and what will allow them to work most
efficiently.
Here's an example of a simple chunk of template using first an in-line
style (Apache::ASP in this case) and then a mini-language style
(Template Toolkit). This code fetches an object and displays some
properties of it. The data structures used are identical in both
examples. First Apache::ASP:
<% my $product = Product->load('sku' => 'bar1234'); %>
<% if ($product->isbn) { %>
It's a book!
<% } else { %>
It's NOT a book!
<% } %>
<% foreach my $item (@{$product->related}) { %>
You might also enjoy <% $item->name %>.
<% } %>
And now Template Toolkit:
[% USE product(sku=bar1234) %]
[% IF product.isbn %]
It's a book!
[% ELSE %]
It's NOT a book!
[% END %]
[% FOREACH item = product.related %]
You might also enjoy [% item.name %].
[% END %]
There is a third approach, based on parsing an HTML document into a
DOM tree and then manipulating the contents of the nodes. The only
module using this approach is HTML_Tree. The idea is similar to using
a mini-language, but it doesn't require any non-standard HTML tags and
it doesn't embed any logic about loops or conditionals in the template
itself. This is nice because it means your templates are valid HTML
documents that can be viewed in a browser and worked with in most
standard HTML tools. It also means people working with the templates
can put placeholder data in them for testing and it will simply be
replaced when the template is used. This preview ability only breaks
down when you need an if/else type construct in the template. In that
situation, both the "if" and "else" chunks of HTML would show up when
previewing.
=head2 Parsers and Caching
The parsers for these templating systems are implemented in one of
three ways: they parse the template every time ("repeated parse"),
they parse it and cache the resulting parse tree ("cached parse
tree"), or they parse it, convert it to Perl code, and compile it
("compiled").
Systems that compile templates to Perl take advantage of Perl's
powerful runtime code evaluation capabilities. They examine the
template, generate a chunk of Perl code from it, and C<eval> the
generated code. After that, subsequent requests for the template can
be handled by running the compiled bytecode in memory. The complexity
of the parsing and code generation steps varies based on the number of
bells and whistles the system provides beyond straight in-line Perl
statements.
Compiling to Perl and then to Perl bytecode is slow on the first hit
but provides excellent performance once the template has been
compiled, since the template becomes a Perl subroutine call. This is
the same approach used by systems like JSP (Java ServerPages). It is
most effective in environments with a long-running Perl interpreter,
like mod_perl.
HTML::Template, HTML_Tree, and the 2.0 beta release of Embperl all use
a cached parse tree approach. They parse templates into their
respective internal data structures and then keep the parsed structure
for each processed template in memory. This is similar to the
compiled Perl approach in terms of performance and memory
requirements, but does not actually involve Perl code generation and
thus doesn't require an C<eval> step. Which way is faster, caching
the parse tree or compiling? It's hard to objectively measure, but
anecdotal evidence seems to support compilation. Template Toolkit
used a cached parse tree approach for version 1, but switched to a
compilation approach for version 2 after tests showed it to offer a
significant speed increase. However, as will be discussed later,
either approach is more than fast enough.
In contrast to this, a repeated parse approach may sound very slow.
However, it can be pretty fast if the tokens being parsed for are
simple enough. Systems using this approach generally use very simple
tokens, which allows them to use fast and simple parsers.
Why would you ever use a system with this approach if compilation has
better performance? Well, in an environment without a persistent Perl
interpreter like vanilla CGI this can actually be faster than a
compiled approach since the startup cost is lower. The caching of
Perl bytecode done by compilation systems is useless when the Perl
interpreter doesn't stick around for more than one request.
There are other reasons too. Compiled Perl code takes up a lot of
memory. If you have many unique templates, they can add up fast.
Imagine how much RAM it would take up if every page that used
server-side includes (SSI) had to stay in memory after it had been
accessed. (Don't worry, the C<Apache::SSI> module doesn't use
compilation so it doesn't have this problem.)
=head2 Application Frameworks vs. Just Templates
Some of the templating tools try to offer a comprehensive solution to
the problems of web development. Others offer just a templating
solution and assume you will fit this together with other modules to
build a complete system.
Some common features offered in the frameworks include:
=head3 URL Mapping
All of the frameworks offer a way to map a URL to a template file. In
addition to simple mappings similar to the handling of static
documents, some offer ways to intercept all requests within a certain
directory for pre-processing, or create an object inheritance scheme
out of the directory structure of a site.
=head3 Session Tracking
Most interactive sites need to use some kind of session tracking to
associate application state data with a user. Some tools make this
very easy by handling all the cookies or URL-munging for you and
letting you simply read and write from an object or hash that contains
the current user's session data. A common approach is to use the
Apache::Session module for storage.
=head3 Output Caching
Caching is the key to good performance in many web systems, and some
of these tools provide user-controlled caching of output. This is one
of the major features of both Mason and AxKit. AxKit can cache at the
page level, while Mason also offers fine-grained caching of components
within the page.
=head3 Form Handling
How will you live without CGI.pm to parse incoming form data? Many of
these tools will do it for you, making it available in a convenient
data structure. Some also validate form input, and even provide
"sticky" form widgets that keep their selected values when
re-displayed or set up default values based on data you provide.
=head3 Debugging
Everyone knows how painful it can be to debug a CGI script.
Templating systems can make it worse, by screwing up Perl's line
numbers with generated code. To help fix the problem they've created,
some offer built-in debugging support, including extra logging, or
integration with the Perl debugger.
If you want to use a system that just does templates but you need some
of these other features and don't feel like implementing them
yourself, there are some tools on CPAN which provide a framework you
can build on. The libservlet distribution, which provides an
interface similar to the Java servlet API, is independent of any
particular templating system. Apache::PageKit and CGI::Application
are other options in this vein, but both of these are currently tied
to HTML::Template. OpenInteract is another framework, this time tied
to Template Toolkit. All of these could be adapted for the "just
templates" module of your choice with fairly minimal effort.
=head1 The Contenders
Okay, now that you know something about what separates these tools
from each other, let's take a look at the top choices for Perl
templating systems. This is not an exhaustive list: I've only
included systems that are currently maintained, well-documented, and
have managed to build up a significant user community. In short, I've
left out a dozen or so less popular systems. At the end of this
section, I'll mention a few systems that aren't as commonly used but
may be worth a look.
=head2 SSI
SSI is the granddaddy of templating systems, and the first one that
many people used since it comes as a standard part of most web
servers. With mod_perl installed, mod_include gains some additional
power. Specifically, it is able to take a new C<#perl> directive
(though only if mod_perl is statically built) which allows for in-line
subroutine calls. It can also efficiently include the output of
Apache::Registry scripts by using the Apache::Include module.
The Apache::SSI module implements the functionality of mod_include
entirely in Perl, including the additional C<#perl> directive. The
main reasons to use it are to post-process the output of another
handler (with Apache::Filter) or to add your own directives. Adding
directives is easy through subclassing. You might be tempted to
implement a complete template processor in this way, by adding loops
and other constructs, but it's probably not worth the trouble with so
many other tools out there.
SSI follows the callback model and is mostly a mini-language, although
you can sneak in bits of Perl code as anonymous subs in C<#perl>
directives. Because SSI uses a repeated parse implementation, it is
safe to use it on large numbers of files without worrying about memory
bloat.
SSI is a great choice for sites with fairly simple templating needs,
especially ones that just want to share some standard headers and
footers between pages. However, you should consider whether or not
your site will eventually need to grow into something with more
flexibility and power before settling on this simple approach.
=head2 HTML::Mason
Mason has been around for a few years now, and has built up a loyal
following. It was originally created as a Perl clone of some of the
most interesting features from Vignette StoryServer, but has since
become it's own unique animal. It comes from a publishing background,
and includes features oriented towards splitting up pages into
re-useable chunks, or "components."
Mason uses in-line Perl with a compilation approach, but has a feature
to help keep the perl code out of the HTML coder's way. Components
(templates) can include a section of Perl at the end of the file which
is wrapped inside a special tag indicating that it should be run
first, before the rest of the template. This allows programmers to
put all the logic for a component down at the bottom away from the
HTML, and then use short in-line Perl snippets in the HTML to insert
values, loop through lists, etc.
Mason is a site development framework, not just a templating tool. It
includes a very handy caching feature that can be used for capturing
the output of components or simply storing data that is expensive to
compute. It is currently the only tool that offers this sort of
caching as a built-in. It also implements an argument parsing scheme
which allows a component to specify the names, types, and default
values that it expects to be passed, either from another component or
from the values passed in the URI query string.
While the documentation mostly demonstrates a callback execution
model, it is possible to use Mason in a pipeline style. This can be
accomplished in various ways, including designating components as
"autohandlers" which run before anything else for requests within a
certain directory structure. An autohandler could do some processing
and set up data for a display template which only includes minimal
in-line Perl. There is also support for an object-oriented site
approach, applying concepts like inheritance to the site directory
structure. For example, the component at /store/book/ might inherit a
standard layout from the component at /store/, but override the
background color and navigation bar. Then /store/music/ can do the
same, with a different color. This can be a very powerful paradigm
for developing large sites.
Mason's approach to debugging is to create "debug files" which run
Mason outside of a web server environment, providing a fake web
request and activating the debugger. This can be helpful if you're
having trouble getting Apache::DB to behave under mod_perl, or using
an execution environment that doesn't provide built-in debugger
support.
Another unique feature is the ability to leave the static text parts
of a large template on disk, and pull them in with a file seek when
needed rather than keeping them in RAM. This exchanges some speed for
a significant savings in memory when dealing with templates that are
mostly static text.
There are many other features in this package, including filtering of
HTML output and a page previewing utility. Session support is not
built-in, but a simple example showing how to integrate with
Apache::Session is included. Mason's feature set can be a bit
overwhelming for newbies, but the high-quality documentation and
helpful user community go a long way.
=head2 HTML::Embperl
Embperl makes its language choice known up front: embedded perl. It
is one of the most popular in-line Perl templating tools and has been
around longer than most of the others. It has a solid reputation for
speed and ease of use.
It is commonly used in a callback style, with Embperl intercepting
URIs and processing the requested file. However, it can optionally be
invoked through a subroutine call from another program, allowing it to
be used in a pipeline style. Templates are compiled to Perl bytecode
and cached.
Embperl has been around long enough to build up an impressive list of
features. It has the ability to run code inside a Safe compartment,
support for automatically cleaning up globals to make mod_perl coding
easier, and extensive debugging tools including the ability to e-mail
errors to an administrator.
The main thing that sets Embperl apart from other in-line Perl systems
is its tight HTML integration. It can recognize C<TABLE> tags and
automatically iterate over them for the length of an array. It
automatically provides sticky form widgets. An array or hash
reference placed at the end of a query string in an C<HREF> or C<SRC>
attribute will be automatically expanded into query string
"name=value" format. C<META HTTP-EQUIV> tags are turned into true HTTP
headers.
Another reason people like Embperl is that it makes some of the common
tasks of web application coding so simple. For example, all form data
is always available just by reading the magic variable %fdat.
Sessions are supported just as easily, by reading and writing to the
magic %udat hash. There is also a hash for storing persistent
application state. HTML-escaping is automatic (though it can be
toggled on and off).
Embperl includes something called EmbperlObject, which allows you to
apply OO concepts to your site hierarchy in a similar way to the
inheritance features mentioned for Mason, above. This is a very
convenient way to code sites with styles that vary by area, and is
worth checking out.
One drawback of older versions of Embperl was the necessity to use
built-in replacements for most of Perl's control structures like "if"
and "foreach" when they are being wrapped around non-Perl sections.
For example:
[$ if ($foo) $]
Looks like a foo!
[$ else $]
Nope, it's a bar.
[$ endif $]
These may seem out of place in a system based around in-line Perl. As
of version 1.2b2, it is possible to use Perl's standard syntax instead:
[$ if ($foo) { $]
Looks like a foo!
[$ } else { $]
Nope, it's a bar.
[$ } $]
At the time of this writing, a new 2.x branch of Embperl is in beta
testing. This includes some interesting features like a more flexible
parsing scheme which can be modified to users' tastes. it also
supports direct use of the Perl debugger on Embperl templates, and
provides performance improvements.
=head2 Apache::AxKit
AxKit is the first mod_perl page generation system to be built from the
ground up around XML. Technically, AxKit itself is not a templating
tool but rather a framework for stringing together different modules
that generate and transform XML data. In fact, it can optionally use
Template Toolkit as an XML transformation language. However, it
deserves coverage here since it is also the home of some templating
tools that are not represented elsewhere.
In its simplest form, AxKit maps XML files to XSL stylesheets which it
can process using commonly available XSLT modules like XML::XSLT or
XML::Sablotron. The rules for mapping a stylesheet to a request are
very flexible, and they can incorporate query strings, cookies, and
other attributes of the request. The idea is that you can use this
feature to handle a wide variety of clients with differing display
capabilities by choosing the right stylesheet.
Recognizing that not everyone is a fan of XSL's somewhat obtuse
syntax, Matt Sergeant has provided an alternate stylesheet language
called XPathScript. XPathScript allows you to write a stylesheet
using text with embedded Perl code. This is similar to the other
embedded Perl templating tools, but the focus is on using the built in
XPath functions for querying an XML document and manipulating the
retrieved data. XPathScript can also be used in a declarative
fashion, specifying the formatting of particular elements in the XML
input. For example this snippet will change all C<E<lt>fooE<gt>> tags
in an XML document to BAR in the output::
<%
$t->{'foo'}{pre} = 'BAR';
$t->{'foo'}{post} = '';
$t->{'foo'}{showtag} = 0;
%>
<%= apply_templates() %>
By using XPathScript's include function (which looks just like SSI),
you can build up libraries of useful transformations that use this
technique.
This is all well and good if you have a bunch of XML files sitting on
a disk somewhere, but what about dynamic content? AxKit handles this
by allowing you to substitute a different data source for the default
file-based one. This can include running some dynamic code on each
request to generate the XML data that will be transformed. The
distribution includes a module for doing this called XSP. XSP is a
language for building an XML DOM using in-line Perl and tag libraries.
The tag libraries are specified as stylesheets which can turn XML tags
into Perl code. This is demonstrated through the included SQL tag
library, which allows you to write an XSP page using XML tags which
will connect to a database, execute queries, and generate an XML
document with the results.
AxKit has some nice performance boosts built into it. It can cache
the full output of a page and serve it as a static file on future
requests. It can also compress output to speed up downloads for
browsers that understand gzip encoding. These can be done with other
systems, but they require you to setup additional software . With
AxKit, you just enable them in the configuration file.
If all of these languages, tag libraries, and stylesheets sound
intimidating to you, AxKit may be overkill for your project. However,
AxKit has the advantage of being built on approved W3C standards, and
many of the skills used in developing for it carry over to other
languages and tools.
=head2 Apache::ASP
Apache::ASP started out as a port of Microsoft's Active Server Pages
technology, and its basic design still follows that model. It uses
in-line Perl with a compilation approach, and provides a set of simple
objects for accessing the request information and formulating a
response. Scripts written for Microsoft's ASP using Perl (via
ActiveState's PerlScript) can usually be run on this system without
changes. (Pages written in VBScript are not supported.)
Like the original ASP, it has hooks for calling specified code when
certain events are triggered, such as the start of a new user session.
It also provides the same easy-to-use state and session management.
Storing and retrieving state data for a whole application or a
specific user is as simple as a single method call. It can even
support user sessions without cookies by munging URLs -- a unique
feature among these systems.
A significant addition that did not come from Microsoft ASP is the XML
and XSLT support. There are two options provided: XMLSubs and XSLT
transforms. XMLSubs is a way of adding custom tags to your pages. It
maps XML tags to your subroutines, so that you can add something like
C<E<lt>site:header page="Page Title" /E<gt>> to your pages and have it
translate into a subroutine call like C<&site::header({title =E<gt>
"Page Title"})>. It can handle processing XML tags with body text as
well.
The XSLT support allows the output of ASP scripts to be filtered
through XSLT for presentation. This allows your ASP scripts to
generate XML data and then format that data with a separate XSL
stylesheet. This support is provided through integration with the
XML::XSLT module.
Apache::ASP provides sticky widgets for forms through the use of the
HTML::FillInForm module. It also has built-in support for removing
extra whitespace from generated output, gzip compressing output (for
browsers that support it), tracking performance using Time::HiRes,
automatically mailing error messages to an administrator, and many
other conveniences and tuning options. This is a mature package which
has evolved to handle real-world problems.
One thing to note about the session and state management in this
system is that it currently only supports clusters through the use of
network filesystems like NFS or SMB. (Joshua Chamas, the module's
author, has reported much better results from Samba file-sharing than
from NFS.) This may be an issue for large-scale server clusters,
which usually rely on a relational database for network storage of
sessions. Support database storage of sessions is planned for a
future release.
=head2 Text::Template
This module has become the de facto standard general purpose
templating module on CPAN. It has an easy interface and thorough
documentation. The examples in the docs show a pipeline execution
style, but it's easy to write a mod_perl handler that directly invokes
templates, allowing a callback style. The module uses in-line Perl.
It has the ability to run the in-line code in a Safe compartment, in
case you are concerned about mistakes in the code crashing your
server.
The module relies on creative uses of in-line code to provide things
that people usually expect from templating tools, like includes. This
can be good or bad. For example, to include a file you could just
call Text::Template::fill_in_file(filename). However, you'll have to
specify the complete file path and nothing will stop you from using
/etc/passwd as the file to be included. Most of the fancier
templating tools have concepts like include paths, which allow you to
specify a list of directories to search for included files. You could
write a subroutine that works this way, and make it available in your
template's namespace, but it's not built in.
Each template is loaded as a separate object. Templates are compiled
to Perl and only parsed the first time they are used. However, to
take full advantage of this caching in a persistent environment like
mod_perl, your program will have to keep track of which templates have
been used, since Text::Template does not have a way of globally
tracking this and returning cached templates when possible.
Text::Template is not tied to HTML, and is just a templating module,
not a web application framework. It is perfectly at home generating
e-mails, PDFs, etc.
=head2 Template Toolkit
One of the more recent additions to the templating scene, Template
Toolkit is a very flexible mini-language system. It has a complete
set of directives for working with data, including loops and
conditionals, and it can be extended in a number of ways. In-line
Perl code can be enabled with a configuration option, but is generally
discouraged. It uses compilation, caching the compiled bytecode in
memory and optionally caching the generated Perl code for templates on
disk. Although it is commonly used in a pipeline style, the included
Apache::Template module allows templates to be invoked directly from
URLs.
Template Toolkit has a large feature set, so we'll only be able cover
some of the highlights here. The TT distribution sets a gold standard
for documentation thoroughness and quality, so it's easy to learn more
if you choose to.
One major difference between TT and other systems is that it provides
simple access to complex data structures through the concept of a dot
operator. This allows people who don't know Perl to access nested
lists and hashes or call object methods. For example, we could pass
in this Perl data structure:
$vars = {
customer => {
name => 'Bubbles',
address => {
city => 'Townsville',
}
}
};
Then we can refer to the nested data in the template:
Hi there, [% customer.name %]!
How are things in [% customer.address.city %]?
This is simpler and more uniform than the equivalent syntax in Perl.
If we pass in an object as part of the data structure, we can use the
same notation to call methods within that object. If you've modeled
your system's data as a set of objects, this can be very convenient.
Templates can define macros and include other templates, and
parameters can be passed to either. Included templates can optionally
localize their variables so that changes made while the included
template is executing do not affect the values of variables in the
larger scope.
There is a filter directive, which can be used for post-processing
output. Uses for this range from simple HTML entity conversion to
automatic truncation (useful for pulldown menus when you want to limit
the size of entries) and printing to STDERR.
TT supports a plugin API, which can be used to add extra capabilities
to your templates. The provided plugins can be broadly organized into
data access and formatting. Standard data access plugins include
modules for accessing XML data or a DBI data source and using that
data within your template. There's a plugin for access to CGI.pm as
well.
Formatting plugins allow you to display things like dates and prices
in a localized style. There's also a table plugin for use in
displaying lists in a multi-column format. These formatting plugins
do a good job of covering the final 5% of data display problems that
often cause people who are using an in-house system to embed a little
bit of HTML in their Perl modules.
In a similar vein, TT includes some nice convenience features for
template writers like eliminating white space around tags and the
ability to change the tag delimiters -- things that may sound a little
esoteric, but can sometimes make templates significantly easier to
work with.
The TT distribution also includes a script called ttree which allows
for processing an entire directory tree of templates. This is useful
for sites that pre-publish their templated pages and serve them
statically. The script checks modification times and only updates
pages that require it, providing a make-like functionality. The
distribution also includes a sample set of template-driven HTML
widgets which can be used to give a consistent look and feel to a
collection of documents.
=head2 HTML::Template
HTML::Template is a popular module among those looking to use a
mini-language rather than in-line Perl. It uses a simple set of tags
which allow looping (even on nested data structures) and conditionals
in addition to basic value insertion. The tags are intentionally
styled to look like HTML tags, which may be useful for some
situations.
As the documentation says, it "does just one thing and it does quickly
and carefully" -- there is no attempt to add application features like
form-handling or session tracking. The module follows a pipeline
execution style. Parsed templates are stored in a Perl data structure
which can be cached in any combination of memory, shared memory (using
IPC::SharedCache), and disk. The documentation is complete and
well-written, with plenty of examples.
You may be wondering how this module is different from Template
Toolkit, the other popular mini-language system. Beyond the obvious
differences in syntax, HTML::Template is faster and simpler, while
Template Toolkit has more advanced features, like plugins and dot
notation. Here's a simple example comparing the syntax:
HTML::Template:
<TMPL_LOOP list>
<a href="<TMPL_VAR url>"><b><TMPL_VAR name></b></A>
</TMPL_LOOP>
Template Toolkit:
[% FOREACH list %]
<a href="[% url %]"><b>[% name %]</a></a>
[% END %]
And now, a few honorable mentions:
=head2 HTML_Tree
As mentioned earlier, HTML Tree uses a fairly unique method of
templating: it loads in an HTML page, parses it to a DOM, and then
programmatically modifies the contents of nodes. This allows it to
use genuine valid HTML documents as templates, something which none of
these other modules can do. The learning curve is a little steeper
than average, but this may be just the thing if you are concerned
about keeping things simple for your HTML coders. Note that the name
is "HTML_Tree", not "HTML::Tree".
=head2 Apache::XPP
XPP is an in-line Perl system that compiles to bytecode. Although it
is a perfectly good implementation, it has little to differentiate it
except for an easy mechanism to define new HTML-like tags which can be
used to replace in-line code in templates.
=head2 ePerl
Possibly the first module to embed Perl code in a text or HTML file,
ePerl is still a viable option in the form of Apache::ePerl. It
caches compiled bytecode in memory to achieve solid performance, and
some people find it refreshingly simple to use.
=head2 CGI::FastTemplate
This module takes a minimalistic approach to templating, which makes
it unusually well suited to use in CGI programs. It parses templates
with a single regular expression and does not support anything in
templates beyond simple variable interpolation. Loops are handled by
including the output of other templates. Unfortunately, this leads to
a Perl coding style that is more confusing than most, and a
proliferation of template files. However, some people swear by this
dirt-simple approach.
=head1 Performance
People always seem to worry about the performance of templating
systems. If you've ever built a large-scale application, you should
have enough perspective on the relative costs of different actions to
know that your templating system is not the first place to look for
performance gains. All of the systems mentioned here have excellent
performance characteristics in persistent execution environments like
mod_perl. Compared to such glacially slow operations as fetching data
from a database or file, the time added by the templating system is
almost negligible.
If you think your templating system is slowing you down, get the
facts: pull out Devel::DProf and see. If one of the tools mentioned
here is at the top of the list for wall clock time used, you should
pat yourself on the back -- you've done a great job tuning your system
and removing bottlenecks! Personally, I have only seen this happen
when I had managed to successfully cache nearly every part of the work
to handle a request except running a template.
However, if you really are in a situation where you need to squeeze a
few extra microseconds out of your page generation time, there are
performance differences between systems. They're pretty much what you
would expect: systems that do the least run the fastest. Using
in-line print() statements is faster than using templates. Using
simple substitution is faster than using in-line Perl code. Using
in-line Perl code is faster than using a mini-language.
The only templating benchmark available at this time is one developed
by Joshua Chamas, author of Apache::ASP. It includes a "hello world"
test, which simply checks how fast each system can spit back those
famous words, and a "hello 2000" test, which exercises the basic
functions used in most dynamic pages. It is available from the
following URL:
http://www.chamas.com/bench/hello.tar.gz
Results from this benchmark currently show SSI, Apache::ASP, and
HTML::Embperl having the best performance of the lot. Not all of the
systems mentioned here are currently included in the test. If your
favorite was missed, you might want to download the benchmark code and
add it. As you can well imagine, benchmarking people's pet projects
is largely a thankless task and Joshua deserves some recognition and
support for this contribution to the community.
=head2 CGI Performance Concerns
If you're running under CGI, you have bigger fish to fry than worrying
about the performance of your templating system. Nevertheless, some
people are stuck with CGI but still want to use a templating system
with reasonable performance. CGI is a tricky situation, since you
have to worry about how much time it will take for Perl to compile the
code for a large templating system on each request. CGI also breaks
the in-memory caching of templates used by most of these systems,
although the slower disk-based caching provided by Mason,
HTML::Template, and Template Toolkit will still work. (HTML::Template
does provide a shared memory cache for templates, which may improve
performance, although shared memory on my Linux system is usually
slower than using the filesystem. Benchmarks and additional
information are welcome.)
Your best performance bet with CGI is to use one of the simpler tools,
like CGI::FastTemplate or Text::Template. They are small and compile
quickly, and CGI::FastTemplate gets an extra boost since it relies on
simple regex parsing and doesn't need to eval any in-line Perl code.
Almost everything else mentioned here will add tenths of seconds to
each page in compilation time alone.
=head1 Matrix
To help you choose a system, I'll summarize the basic characteristics
of the major systems along the decision points I've explained in the
beginning of the article. Keep in mind that in many cases a system
can be used in more than one way, and I've simply shown the dominant
method as seen in the documentation and real world use. You should
not eliminate options based on this chart without reading the more
detailed explanations above.
=for html
<TABLE BORDER=1>
<TR>
<TH><br></TH>
<TH>
Application Framework
</TH>
<TH>
Pipeline or Callback
</TH>
<TH>
Parsing Method
</TH>
<TH>
Language
</TH>
</TR>
<TR>
<TD>
HTML::Mason
</TD>
<TD>
Framework
</TD>
<TD>
Callback
</TD>
<TD>
Compiled
</TD>
<TD>
Perl
</TD>
</TR>
<TR>
<TD>
Template Toolkit
</TD>
<TD>
Just Templates
</TD>
<TD>
Pipeline
</TD>
<TD>
Compiled
</TD>
<TD>
Mini-Language
</TD>
</TR>
<TR>
<TD>
Apache::ASP
</TD>
<TD>
Framework
</TD>
<TD>
Callback
</TD>
<TD>
Compiled
</TD>
<TD>
Perl and XSL
</TD>
</TR>
<TR>
<TD>
HTML::Embperl
</TD>
<TD>
Framework
</TD>
<TD>
Callback
</TD>
<TD>
Compiled
</TD>
<TD>
Perl
</TD>
</TR>
<TR>
<TD>
SSI
</TD>
<TD>
Just Templates
</TD>
<TD>
Callback
</TD>
<TD>
Repeated Parse
</TD>
<TD>
Mini-Language
</TD>
</TR>
<TR>
<TD>
AxKit
</TD>
<TD>
Framework
</TD>
<TD>
Pipeline
</TD>
<TD>
Compiled or Cached Parse Tree
</TD>
<TD>
Perl and XSL and Mini-Language(s)
</TD>
</TR>
<TR>
<TD>
HTML::Template
</TD>
<TD>
Just Templates
</TD>
<TD>
Pipeline
</TD>
<TD>
Cached Parse Tree
</TD>
<TD>
Mini-Language
</TD>
</TR>
<TR>
<TD>
Text::Template
</TD>
<TD>
Just Templates
</TD>
<TD>
Pipeline
</TD>
<TD>
Compiled
</TD>
<TD>
Perl
</TD>
</TR>
</TABLE>
=head1 Updates
These modules are moving targets, and a document like this is bound to
contain some mistakes. Send your corrections to [EMAIL PROTECTED]
Future versions of this document will be announced on the mod_perl
mailing list, and possibly other popular Perl locations as well.
by Perrin Harkins
=head1 Maintainers
Maintainer is the person(s) you should contact with updates,
corrections and patches.
=over
=item *
Perrin Harkins E<lt>perrin (at) elem.comE<gt>.
=back
=head1 Authors
=over
=item *
Perrin Harkins E<lt>perrin (at) elem.comE<gt>.
=back
Only the major authors are listed above. For contributors see the
Changes file.
=cut
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
