mod_perl + multiple Oracle schemas (was RE: Edmund Mergl)
John-- Another thing you may want to look into is just doing an "alter session set current_schema" call at the top of your mod_perl page. This is actually significantly faster than Tim's reauthenticate solution (about 7X, according to my benchmarks). It has become a supported feature as of Oracle 8i. For details on what I did, see http://www.lifespree.com/modperl/ (which is still a total mess right now-- I'll get around to cleaning it up sometime soon, I promise!) cheers, Ed -Original Message- From: John D Groenveld [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 10, 2001 5:10 PM To: Edmund Mergl Cc: [EMAIL PROTECTED] Subject: Re: Edmund Mergl Good to see you alive, well, and still coding Perl. Months ago, about the time of the Perl conference so it may have slipped under everyone's radar, Jeff Horn from U of Wisconsin sent you some patches to Apache::DBI to use Oracle 8's re-authenticate function instead of creating and caching a separate Oracle connection for each user. Did you decide whether to incorporate them or to suggest another module name for him to use? I wasn't able to participate in the discussion at the time, but I now have need for that functionality. I don't know if Jeff Horn is still around, but I'll track him down if necessary and offer to work on it. Also, I sent you a small patch to fix Apache::DBI warnings under Perl5.6. I hate to be a pest, but I'm rolling out software where the installation procedure requires the user to fetch Perl from Active State and Apache::DBI from CPAN. I'd rather not ship my own version of yours or any CPAN module. Thanks, John [EMAIL PROTECTED]
getting rid of multiple identical http requests (bad users double-clicking)
Does anyone out there have a clean, happy solution to the problem of users jamming on links & buttons? Analyzing our access logs, it is clear that it's relatively common for users to click 2,3,4+ times on a link if it doesn't come up right away. This not good for the system for obvious reasons. I can think of a few ways around this, but I was wondering if anyone else had come up with anything. Here are the avenues I'm exploring: 1. Implementing JavaScript disabling on the client side so that links become 'click-once' links. 2. Implement an MD5 hash of the request and store it on the server (e.g. in a MySQL server). When a new request comes in, check the MySQL server to see whether it matches an existing request and disallow as necessary. There might be some sort of timeout mechanism here, e.g. don't allow identical requests within the span of the last 20 seconds. Has anyone else thought about this? cheers, Ed
RE: the edge of chaos
A few thoughts: In analyzing a few spikes on our site in the last few days, a clear pattern has emerged: the database spikes, and the database spikes induce a corresponding spike on the mod_perl server about 2-6 minutes later(because mod_perl requests start queuing up). This is exacerbated by the fact that as the site slows down, folks start double and triple-clicking on links and buttons, which of course just causes things to get much worse. This has a few ramifications. If your pages are not homogeneous in database usage (i.e., some pages are much heavier than others), then throttling by number of connections or throttling based on webserver load doesn't help that much. You need to throttle based on database server load. This requires some sort of mechanism whereby the webserver can sample the load on the database server and throttle accordingly. Currently, we just mount a common NFS fileserver, sample every minute, and restart the webserver if db load is too high, which works OK. The best course of action, though, is to tune your database, homogenize your pages, and buy a bigger box, which we're doing. -Ed -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Perrin Harkins Sent: Thursday, January 04, 2001 6:38 PM To: Justin Cc: [EMAIL PROTECTED] Subject: Re: the edge of chaos Justin wrote: > Thanks for the links! But. I wasnt sure what in the first link > was useful for this problem, and, the vacuum bots discussion > is really a different topic. > I'm not talking of vacuum bot load. This is real world load. > > Practical experiments (ok - the live site :) convinced me that > the well recommended modperl setup of fe/be suffer from failure > and much wasted page production when load rises just a little > above *maximum sustainable throughput* .. The fact that mod_proxy doesn't disconnect from the backend server when the client goes away is definitely a problem. I remember some discussion about this before but I don't think there was a solution for it. I think Vivek was correct in pointing out that your ultimate problem is the fact that your system is not big enough for the load you're getting. If you can't upgrade your system to safely handle the load, one approach is to send some people away when the server gets too busy and provide decent service to the ones you do allow through. You can try lowering MaxClients on the proxy to help with this. Then any requests going over that limit will get queued by the OS and you'll never see them if the person on the other end gets tired of waiting and cancels. It's tricky though, because you don't want a bunch of slow clients to tie up all of your proxy processes. It's easy to adapt the existing mod_perl throttling handlers to send a short static "too busy" page when there are more than a certain number of concurrent requests on the site. Better to do this on the proxy side though, so maybe mod_throttle could do it for you. - Perrin
showing mod_perl execute time in access_log
quick, obvious trick:
This is a trivial modification of Doug's original Apache::TimeIt script that
allows you to very precisely show the Apache execute time of the page.
This is particularly useful if you want to know which pages of your site you
could optimize.
Here's a question, though: does anyone know an easy way of measuring how
long apache keeps a socket to the client open, assuming that KeepAlive has
been turned off? This is relevant because I want to know how long on average
it is taking clients to receive certain pages in my application. I know that
I can approximately calculate it from bandwidth, but I would expect the
actual number to vary wildly throughout a given day due to Internet
congestion.
cheers,
Ed
---
package AccessTimer;
# USAGE:
# Just put the following line into your .conf file:
#
# PerlFixupHandler AccessTimer
#
# and use a custom Apache log (this logging piece is not at all
mod_perl-based...
# see http://httpd.apache.org/docs/mod/mod_log_config.html)
#
# CustomLog /path/to/your/log "%h %l %u %t \"%r\" %>s %b %{ELAPSED}e"
#
use strict;
use Apache::Constants qw(:common);
use Time::HiRes qw(gettimeofday tv_interval);
use vars qw($begin);
sub handler {
my $r = shift;
$begin = [gettimeofday];
$r->push_handlers(PerlLogHandler=>\&log);
return OK;
}
sub log {
my $r = shift;
my $elapsed = tv_interval($begin);
$r->subprocess_env('ELAPSED' => "$elapsed");
return DECLINED;
}
1;
RE: Mod_perl tutorials
My two cents-- I really like the look of the take23 site as well, and I would be happy as a clam if we could get modperl.org. I'd even be willing to chip in some (money/time/effort) to see whether we could get modperl.org. More than that, though, I think that I would really like to see take23 in large measure replace the current perl.apache.org. I remember the first time I looked at perl.apache.org, it was not at all clear to me that I could build a fast database-backed web application using mod_perl. In contrast, when you click on PHP from www.apache.org, you are taken directly to a site that gives you the sense that there is a strong, vibrant community around php. (BTW, I also like the look and feel of take23 significantly more than php). Anyways, those are my own biases. The final bias is that the advocacy site should be hosted someplace _fast_; one of the reasons I initially avoided PHP was that their _site_ was dog slow, and I associated that with PHP being dog slow. Anyways, take23 is very fast for now. cheers, Ed
RE: Article idea: mod_perl + JSP
I've been thinking about this quite a bit recently. I agree with Gunther-- what is more important is not really the language that you use, but the high-level application framework you have built for yourself and how you use it. This is because most of the essential elements of any framework can be duplicated in any reasonably powerful programming language (mod_perl, java, tcl, even VBScript/VB to some extent). That said, my own experience and benchmarking showed that mod_perl is the best of these architectures for building extremely high-quality, reliable, _complicated_ 3-tier Internet applications in which the prototyping and release cycle are very highly compressed, because I can write the most high-quality, high-speed code per unit time in Perl. By 3-tier apps, I mean apps consisting of a browser, the app server (mod_perl), and an RDBMS. Perl doesn't have much support in the way of n-tier apps, which is why I find Nathan's question interesting, and why I have been thinking about it recently. From some recent experiences with using SOAP to integrate with an outside vendor, I believe that it is possible to create a best-of-breed n-tier solution using Perl as the glue layer. For those of you who don't know what SOAP is, it's essentially RPC over XML, and allows any app to talk to any other app in a standard, XML-based format. Go to http://www.soaplite.com for a very clean implementation of SOAP for Perl. To continue-- there are a few reasons that you might want to use Java as a component of a mod_perl app: -There are sometimes pre-written components for Java that you'd like to use because a vendor has written pre-specified hooks for Java. This could also be the case in which you have to integrate with any legacy systems. -Java has much better support for threading, and therefore in many cases makes a much better server (the simple example for this is a chat server). -Because of Java's threads, it can pool transactions resources (e.g. databases) better, and may therefore be more efficient in places where resources are tightly constrained, _especially_ if the database is queried relatively infrequently. For similar reasons, you might want to use a VB component in your application, etc. SOAP makes that possible. The point here is that I think that an awful lot of folks out there have straitjacketed themselves into thinking that if there's a complicated problem that needs to be solved, and there's a piece of that problem is best done in Java, then we ought to write the whole thing in Java. What I'm saying is that that's not necessarily true-- that it's actually possible to write best-of-breed solutions by introducing a communications-layer abstraction that enables you to build a clean n-tier architecture. CORBA promised this, but was sufficently difficult to implement because it has not (to my knowledge) gained very wide acceptance in the Perl community. Also, the major ORBs (IONA, Visigenic) have largely overlooked creating Perl bindings for theri apps. SOAP, however, makes distributed computing extremely easy and _very_ clean, and I think that it could change the way that people think about building complicated, high-quality applications in an extremely compressed timecycle. Using SOAP actually opens a number of other possibilities that don't require thinking outside of mod_perl, too. For example, one of the big selling points of Java is that it allows horizontal partitioning of classes on different machines. Using SOAP, you can actually partition your _perl_ logic so that different pieces run on different machines; or, you can write a component in Perl that is subsequently called by a Java component. OK, enough of my rambling... cheers, Ed -Original Message- From: Gunther Birznieks [mailto:[EMAIL PROTECTED]] Sent: Tuesday, December 12, 2000 11:42 PM To: Chris Winters; Nathan Torkington Cc: [EMAIL PROTECTED] Subject: Re: Article idea: mod_perl + JSP At 11:11 PM 12/12/2000 -0500, Chris Winters wrote: >* Nathan Torkington ([EMAIL PROTECTED]) [001212 22:09]: > > > > Anyone run such an installation? Anyone want to write about their > > setup and experiences? > > There are projects where we use mod_perl handlers for stuff like prototyping auth but then use servlets/JSPs for the app. But I believe that's too shallow for what Nat wants. We're a small shop of primarily fairly senior developers (at least several years experience in the languages we like to use)... and we've actually found that the Java web and the Perl web projects we've delivered on aren't necessarily THAT far off in project delivery time than some Perl people would have you believe. Of course, we have a toolkit that we use to develop apps in both Perl and Java which helps, but it's still interesting that business logic for people experienced in the language of their choice isn't that bad in terms of delivery time. Of course, maintenance is another issue. I'll probably get some flak for the above statement about Perl vs Jav
Apache::Session benchmarks
FYI-- here are some Apache::Session benchmark results. As with all benchmarks, this may not be applicable to you. Basically, though, the results show that you really ought to use a database to back your session stores if you run a high-volume site. Benchmark: This benchmark measures the time taken to do a create/read for 1000 sessions. It does not destroy sessions, i.e. it assumes a user base that browses around arbitrarily and then just leaves (i.e. does not log out, and so session cleanup can't easily be done). RESULTS: I tested the following configurations: Apache::Session::MySQL - Dual-PIII-600/512MB/Linux 2.2.14SMP: Running both the httpd and mysqld servers on this server. Average benchtime: 2.21 seconds (consistent) Apache::Session::Oracle - Ran the httpd on the dual-PIII-600/512MB/Linux 2.2.14SMP, running Oracle on a separate dual PIII-500/1G (RH Linux 6.2). Average benchtime: 3.1 seconds (consistent). (ping time between the servers: ~3ms) Apache::Session::File - Dual-PIII-600/512MB/Linux 2.2.14SMP: Ran 4 times. First time: ~2.2s. Second time: ~5.0s. Third time: ~8.4s. Fourth time: ~12.2s. Apache::Session::DB_File - Dual-PIII-600/512MB/Linux 2.2.14SMP: Ran 4 times. First time: ~20.0s. Second time: ~20.8s. Third time: ~21.9s. Fourth time: ~23.2s. The actual benchmarking code can be found at http://www.lifespree.com/modperl/ (warning - the site is in a terrible state right now, mostly a scratchpad for various techniques & benchmarks) Question: does anyone know how to pre-specify the _session_id for the session, rather than allowing Apache::Session to set it and read it? I saw some posts about it a while back, but no code... cheers, Ed
[JOB] mod_perl folks wanted in Boston - athenahealth.com
In the spirit of all of this talk about certification, demand for mod_perl programmers, etc., I'd just like to say that I'm looking for programmers. More to the point, I'm looking for kickass folks who just happen to know mod_perl. If you know mod_perl very well, great, but generally speaking, I'm looking for folks who are just kickass hackers, know that they are kickass hackers, and are willing to do anything to drive a problem to extinction. Experience with mod_perl, Linux, Oracle, Solaris, Java, XML/SOAP, MQ Series, transaction brokers, systems administration, NT, DHTML, JavaScript, etc. etc. are all Good Things. But basically, we're looking for folks who are itching to prove themselves and have some sort of history that indicates that they can do it. As a backdrop: we just raised $30 million, and we were the top story in the latest Red Herring VC Dealflow. http://www.redherring.com/vc/2000/1206/vc-ltr-dealflow120600.html As you have probably gathered by now from my posts about the Scaling mod_perl page (http://www.lifespree.com/modperl/- soon to be folded into the Guide), I'm currently starting up a scaling mod_perl project, and I have a lot of money and stock options to burn on good people and interesting toys. If you're interested, send me a private email & a resume and we'll talk. Unfortunately, you sort of have to be in the Boston area (or willing to move) to make this work. cheers, Ed - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [ANNOUNCE] new site: scaling mod_perl will be movin to the Guide
I've gotten in touch with Stas, and the 'scaling mod_perl' site will eventually be folded into the Guide. woohoo! I'm going to spend several weeks fleshing it out and cleaning it up before it goes in, though. -Ed -Original Message- From: Perrin Harkins [mailto:[EMAIL PROTECTED]] Sent: Friday, December 08, 2000 12:36 PM To: Ed Park; [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: [ANNOUNCE] new site: scaling mod_perl (+tool: mod_perl + DBD::Oracle) > The enterprise mod_perl architectures idea that I posted earlier has evolved > into a slightly modified idea: a 'scaling mod_perl' site: > http://www.lifespree.com/modperl. > > The point of this site will be to talk about & synthesize techniques for > scaling, monitoring, and profiling large, complicated mod_perl > architectures. No offense, but the content you have here looks really well suited to be part of the Guide. It would fit nicely into the performance section. Making it a separate site kind of fragments the documentation. > So far, I've written up a basic scaling framework, and I've posted a > particular development profiling tool that we wrote to capture, time, and > explain all SQL select queries that occur on a particular page of a mod_perl > + DBD::Oracle application: > -http://www.lifespree.com/modperl/explain_dbitracelog.pl > -http://www.lifespree.com/modperl/DBD-Oracle-1.06-perfhack.tar.gz Take a look at DBIx::Profile as well. > 1. Performance benchmarking code. In particular, I'm looking for tools that > can read in an apache log, play it back realtime (by looking at the time > between requests in the apache log), and simulate slow & simultaneous > connections. I've started writing my own, but it would be cool if something > else out there existed. The mod_backhand project was developing a tool like this called Daquiri. > If folks could just send me pointers to various caching > modules and code, I'll test them in a uniform environment and let folks know > what I come up with. There are a bunch of discussions about this in the archives, including one this week. Joshua Chamas did some benchmarking on a dbm-based approach recently. - Perrin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[ANNOUNCE] new site: scaling mod_perl (+tool: mod_perl + DBD::Oracle)
The enterprise mod_perl architectures idea that I posted earlier has evolved into a slightly modified idea: a 'scaling mod_perl' site: http://www.lifespree.com/modperl. The point of this site will be to talk about & synthesize techniques for scaling, monitoring, and profiling large, complicated mod_perl architectures. So far, I've written up a basic scaling framework, and I've posted a particular development profiling tool that we wrote to capture, time, and explain all SQL select queries that occur on a particular page of a mod_perl + DBD::Oracle application: -http://www.lifespree.com/modperl/explain_dbitracelog.pl -http://www.lifespree.com/modperl/DBD-Oracle-1.06-perfhack.tar.gz Currently, I'm soliciting thoughts and code on the following subjects in particular: 1. Performance benchmarking code. In particular, I'm looking for tools that can read in an apache log, play it back realtime (by looking at the time between requests in the apache log), and simulate slow & simultaneous connections. I've started writing my own, but it would be cool if something else out there existed. 2. Caching techniques. I know that this is a topic that has been somewhat beaten to a pulp on this list, but it keeps coming up, and I don't know of any place where the current best thinking on the subject has been synthesized. I haven't used any caching techniques yet myself, but I intend to begin caching data at the mod_perl tier in the next version of my application, so I have a very good incentive to synthesize and benchmark various techniques. If folks could just send me pointers to various caching modules and code, I'll test them in a uniform environment and let folks know what I come up with. Or, if someone has already done all that work of testing, I'd appreciate if you could point me to the results. I'd still like to run my own tests, though. If folks could point me towards resources/code for these topics (as well as any other topics you think might be relevant to the site), please let me know. I'm offering to do the legwork required to actually test, benchmark, and synthesize all of this stuff, and publish it on the page. I'm also still interested in actually talking with various folks. If anyone who has been through some significant mod_perl scaling exercise would like to chat for 15-30 minutes to swap war stories or tactical plans, I'd love to talk with you; send me a private email. cheers, Ed - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[OT] RE: Help needed with MAP expression
The point of this function is to right-align numbers in table-data cells and
keep everything else left-aligned. Note that this is what Excel does by
default (if you type in a number in Excel, it aligns to the right; if you
type in a string, it aligns to the left).
Technically, it should be use in the context of an arrayref that you are
transforming into an HTML table, e.g.:
use CGI qw(:all);
$_ = ['1','abc','2.34'];
print join "", map(/^[\.\d]+$/ ? td({-align=>'right'}, $_) : td($_), @$_);
A useful and clever piece of code, that. But the author probably should have
commented it. :)
cheers,
Ed
-Original Message-
From: bari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 07, 2000 12:47 PM
To: [EMAIL PROTECTED]
Subject: Help needed with MAP expression
Hi there,
Can any one help me what this MAP function does...
map(/^[\.\d]+$/ ? td({-align=>'right'}, $_) : td($_), @$_)
I am really confused by this one... your help would be appreciated..
Thank You,
- Bari
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: connect_cached, mod_perl && Oracle connection pooling
Hey--
I know that this is mad late, but this caught my eye, and it doesn't look
like anyone has responded since then.
For anyone else-- if you've even been in a situation where you've wanted to
create persistent DBI connections to multiple Oracle schemas, read on.
In short, here's the solution for that particular problem: there is a
completely undocumented function that is particular to DBD::Oracle that
allows you change your default schema within a particular database
connection.
See http://www.geocrawler.com/archives/3/183/2000/4/0/3652431/
In particular, if you look at the official DBD::Oracle install-test
directory (you can see it in /tmp/cpan/DBD-Oracle-1.03/t on titan), there's
a file called t/reauth.t. Here's a small chunk of code from that:
ok(0, ($dbh->selectrow_array("SELECT USER FROM DUAL"))[0] eq $uid1 );
ok(0, $dbh->func($dbuser_2, '', 'reauthenticate'));
ok(0, ($dbh->selectrow_array("SELECT USER FROM DUAL"))[0] eq $uid2 );
Early tests indicate that reauthentication is not a very expensive function
at all. We are currently testing this approach in development, and plan to
put it into production in the near future. If you use it, lemme know how it
works for you.
hope this helps,
Ed
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Saturday, September 23, 2000 6:09 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: connect_cached, mod_perl && Oracle connection pooling
DBI's connect_cached has been "new" for quite sometime now and has been
labeled with "exact behavior of this method is liable to change" since
it first showed up last year.
In November last year, Tim clarified his intentions for connect_cached
and suggested that DBI::ProxyServer be enhanced to provide a pool from
which connections can be checked in and out (or something like that).
Well, I'm now looking at possibly having a multiplicity of connect
strings in a mod_perl environment. So Apache::DBI doesn't sound
suitable, I don't want every child to maintain connections nailed up for
every connect string (20 apache children * 20 connect strings = 400
nailed up connections, yowza!). At any given time, the processing
happening in the mod_perl apache child process will only need one of
those connect strings. Persistence connections are important just
because of the expense of setting up the Oracle connection. So, I'm
wondering what folks think of non-persistent connections between
mod_perl and the dbiproxy but persistent connections with connect_cached
between dbiproxy and Oracle... does this make sense? I was thinking
it'd be cool to be able to specify how many of each connections should
be maintained in the pool. Is anybody doing this and care to share
their experiences with it?
thanks,
-Ian
--
Salon Internet http://www.salon.com/
Manager, Software and Systems "Livin' La Vida Unix!"
Ian Kallen <[EMAIL PROTECTED]> / AIM: iankallen / Fax: (415) 354-3326
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: eval statements in mod_perl
This was a problem that I had when I was first starting out with mod_perl;
i.e., it wouldn't work the first or second times through, and then it would
magically start working.
This was always caused for me by a syntax error in a library file. In your
case, it could be caused by a syntax error in a library file used somewhere
in your eval'd code. I highly suggest running
> perl -c
on all of your library files to check them for valid syntax. If all of your
library files are in the same directory,
> perl -c *
will work as well.
I'm not certain for the technical reason for this, but I believe it has
something to do with the fact that syntax errors in the libraries are not in
and of themselves considered a fatal condition for loading libraries in
mod_perl, so the second or third time around the persistent mod_perl process
thinks that it has successfully loaded the library. Obviously, some
functions in that library won't work, but you won't know that unless you
actually use them. Someone else might be able to shed more light on this.
good luck,
Ed
-Original Message-
From: Gunther Birznieks [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 07, 2000 3:38 AM
To: Hill, David T - Belo Corporate; '[EMAIL PROTECTED]'
Subject: Re: eval statements in mod_perl
Without knowing your whole program, this could be a variety of logic
problems leading to this code. For example, perhaps $build{$nkey} is a
totally bogus value the first 2 times and hence your $evalcode is also
bogus the first two times -- and it's not a problem of eval at all!
This is unclear for the snippet.
At 10:52 AM 12/6/2000 -0600, Hill, David T - Belo Corporate wrote:
>Howdy,
> I am running mod_perl and have created a handler that serves all
the
>pages for our intranet. In this handler I load perl programs from file
into
>a string and run eval on the string (not in a block). The problem is that
>for any session the code doesn't work the first or second time, then it
>works fine. Is this a caching issue or compile-time vs. run-time issues?
I
>am sure this is a simple fix. What am I missing?
>
> Here is the nasty part (don't throw stones :) So that we can
>develop, I put the eval in a loop that tries it until it returns true or
>runs 3 times. I can't obviously leave it this way. Any suggestions? Here
>is the relevant chunk of code:
>
> # Expect perl code. Run an eval on the code and execute it.
> my $evalcode = "";
> my $line = "";
> open (EVALFILE, $build{"$nkey"});
> while ($line = ) {
> $evalcode .= $line;
> }
> my $evalresult = 0;
> my $counter=0;
>
>#
> # Temporary measure to overcome caching issue, try
to
>#
> # run the eval code 3 times to get a true return.
>#
>
>#
> until (($evalresult) || ($counter eq 3)) {
> $evalresult = eval $evalcode;
> $counter++;
> }
> $pageHash{"Retries"} = $counter if $counter > 1;
> $r->print($@) if $@;
> close (EVALFILE);
>
>I appreciate any and all constructive comments.
>
>-
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
__
Gunther Birznieks ([EMAIL PROTECTED])
eXtropia - The Web Technology Company
http://www.extropia.com/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
enterprise mod_perl architectures
I've been using mod_perl for two years, and I'm currently particularly interested in: 1) Definitively establishing mod_perl as a credible player in the enterprise space. 2) Discussing enterprise-level architecture considerations, performance benchmarks, development methodologies, etc. By "enterprise", I mean folks who have serious money to throw at mod_perl-- folks who have racks of quad-Xeon, 4GB servers at Exodus serving millions of pages per day, and who have been asked (or are being asked) by the company to make their application infinitely scalable. I know that there are several of you out there. As a bit of background, I started out using mod_perl/Embperl because pound for pound, it allowed me to develop a high-quality production application in the shortest amount of time on a stable platform. At that point, WebLogic and EJB were at their version .9 spec (and were highly unstable besides), PHP did not have very rich third-party module support. I have continued to use mod_perl because of its outstanding performance, stability, and architectural characteristics-- in the two years that our site has been up in production, I have had exactly zero problems that could be traced to instabilities in mod_perl/Embperl. During those two years, we grew from an angel-backed garage company to a company that is now backed by Draper Fisher Jurvetson, Venrock, Cardinal Health Partners, and (most recently) Oak Investment Partners. Guy Kawasaki (CEO of garage.com) was once quoted as saying that the amount of pressure increases with the square of the amount money that you have, and I have certainly found that to true. We are now at the hockey stick of growth. Currently, our application serves 300,000 dynamic page requests per day, each of which executes on the average of about 20 SQL statements against an Oracle RDBMS running on Linux. By the end of next year, the application needs to be able to support at least 8 million dynamic page requests per day; and the year after that, 30+ million per day. The project that I propose is simple and concrete: create an open forum in which all of the folks who are currently undergoing the same growing pains that we are, or who have been through them already, or who are otherwise interested in the project, to address the following questions, specifically as they relate to mod_perl + Oracle (or perhaps some other RDBMS): 1) What are the software building blocks that you use? I'm interested specifically in mod_perl + Oracle, but I'd also welcome questions about mod_perl + apache. 2) What is the logical scaling architecture that you use? i.e., how have you architected your mod_perl application so that it will scale linearly with additional hardware? 3) What is the physical scaling and high-availability architecture that you have used? e.g., what is the best way to load-balance across a set of mod_perl servers? Do folks use LVS, or mod_proxy, or F5 BigIPs, or Arrowpoint switches... etc; what hardware are folks using? Dell, Compaq, VA Linux, Penguin, etc.-- what vendor did you choose, and why? Have you had any problems with the vendor? Are folks running mod_perl in production on Linux, or have folks found BSD or Solaris to be better servers, and for what reasons? 4) What tools do folks use to monitor your mod_perl servers? For example, I have written apache modules that simply time how long it takes to execute a page, and a script that does a quick https 'GET' from all production servers to make certain they are up. 5) What tools do folks use to profile the performance of the application? I have also written scripts that essentially do an 'explain plan' in Oracle on every query executed for a given pageview. (This was done very painfully, by hacking dbdimp.c... in retrospect, there may have been an easier way of hacking it at the DBD:: layer, but I wasn't sufficently familiar with the mechanics of XS to do that). I have also tried DProf, but I was unable to get it to work in any meaningful way (don't remember exactly why now) 5) How do you estimate server capacity? For example, has anyone used tools that proxy all requests for a given day and play them back, or take the apache logs for a server for a given day and play them back at varying speeds, simulating varying bandwidth? 6) etc. As you can probably tell, a good deal of this will be mod_perl-specific, and a good deal will simply be a forum for folks to talk _from experience_ about the relative pros and cons of various production architectures that may or may not have anything to do with mod_perl. This is a conversation that may in places go off-topic, and in some places -way- off-topic. However, it is a conversation that I think should ultimately benefit the mod_perl community because it should eventually provide examples of mod_perl architectures, in production, that scale to tens or hundreds of millions of requests per day, and should help to ease some of the tensions that are generated when top management and VCs come knoc
RE: setting LD_LIBRARY_PATH via PerlSetEnv does not work
I ran into this exact same problem this weekend using: -GNU ld 2.9.1 -DBD::Oracle 1.06 -DBI 1.14 -RH Linux 6.0 -Oracle 8i Here's another, cleaner (I think) solution to your problem: after running perl Makefile.PL, modify the resulting Makefile as follows: 1. search for the line LD_RUN_PATH= 2. replace it with LD_RUN_PATH=(my_oracle_home)/lib (my_oracle_home) is, of course, the home path to your oracle installation. In particular, the file libclntsh.so.8.0 should exist in that directory. (If you use cpan, the build directory for DBD::Oracle should be in ~/.cpan/build/DBD-Oracle-1.06/ if you're logged in as root.) Then, just type make install, and all should go well. FYI, setting LD_RUN_PATH has the effect of hard-coding the path to (my_oracle_home)/lib in the resulting Oracle.so file generated by the DBD::Oracle so that at run-time, it doesn't have to go searching through LD_LIBRARY_PATH or the default directories used by ld. The reason I think this is cleaner is because this way, the Oracle directory is not hardcoded globally into everyone's link paths, which is what ldconfig does. For more information, check out the GNU man page on ld: http://www.gnu.org/manual/ld-2.9.1/html_mono/ld.html or an essay on LD_LIBRARY_PATH: http://www.visi.com/~barr/ldpath.html cheers, Ed -Original Message- From: Stas Bekman [mailto:[EMAIL PROTECTED]] Sent: Monday, August 21, 2000 6:51 AM To: Richard Chen Cc: Yann Ramin; [EMAIL PROTECTED] Subject: Re: setting LD_LIBRARY_PATH via PerlSetEnv does not work On Mon, 21 Aug 2000, Richard Chen wrote: > It worked like a charm! If PerlSetEnv could not do it, I think > this should be documented in the guide. I could not find any mention done. thanks for the tip! > about ldconfig in the modperl guide. May be I missed it somehow. > > The procedure on linux is very simple: > # echo $ORACLE_HOME/lib >> /etc/ld.so.conf > # ldconfig > > Thanks > > Richard > > On Sun, Aug 20, 2000 at 08:11:50PM -0700, Yann Ramin wrote: > > As far as FreeBSD goes, LD_LIBRARY_PATH is not searched for setuid > > programs (aka, Apache). This isn't a problem for CGIs since they don't > > do a setuid (and are forked off), but Apache does, and mod_perl is in > > Apache. I think thats right anyway :) > > > > You could solve this globaly by running ldconfig (I assume Linux has it, > > FreeBSD does). You'd be looking for: > > > > ldconfig -m > > > > Hope that helps. > > > > Yann > > > > Richard Chen wrote: > > > > > > This is a redhat linux 6.2 box with perl 5.005_03, Apache 1.3.12, > > > mod_perl 1.24, DBD::Oracle 1.06, DBI 1.14 and oracle 8.1.6. > > > For some odd reason, in order to use DBI, I have to set > > > LD_LIBRARY_PATH first. I don't think I needed to do this when I > > > used oracle 7. This is fine on the command line because > > > I can set it in the shell environment. For cgi scripts, > > > the problem is also solved by using apache SetEnv directive. However, > > > this trick does not work under modperl. I had tried PerlSetEnv > > > to no avail. The message is the same as if the LD_LIBRARY_PATH is not set: > > > > > > install_driver(Oracle) failed: Can't load > > > '/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/Oracle/Oracle.so' for module DBD::Oracle: > > > libclntsh.so.8.0: cannot open shared object file: No such file or directory at > > > /usr/lib/perl5/5.00503/i386-linux/DynaLoader.pm line 169. at (eval 27) line 3 Perhaps a required shared > > > library or dll isn't installed where expected at /usr/local/apache/perl/tmp.pl line 11 > > > > > > Here is the section defining LD_LIBRARY_PATH under Apache::Registry: > > > > > > PerlModule Apache::Registry > > > Alias /perl/ /usr/local/apache/perl/ > > > > > > PerlSetEnv LD_LIBRARY_PATH /u01/app/oracle/product/8.1.6/lib > > > SetHandler perl-script > > > PerlHandler Apache::Registry > > > Options ExecCGI > > > PerlSendHeader On > > > allow from all > > > > > > > > > Does anyone know why PerlSetEnv does not work in this case? > > > How come SetEnv works for cgi scripts? What is the work around? > > > > > > Thanks for any info. > > > > > > Richard > > > > -- > > > > > > Yann Ramin [EMAIL PROTECTED] > > Atrus Trivalie Productions www.redshift.com/~yramin > > Monterey High ITwww.montereyhigh.com > > ICQ 46805627 > > AIM oddatrus > > Marina, CA > > > > IRM Developer Network Toaster Developer > > SNTS Developer KLevel Developer > > > > (yes, this .signature is way too big) > > > > "All cats die. Socrates is dead. Therefore Socrates is a cat." > > - The Logician > > > > THE STORY OF CREATION > > > > In the beginning there was data. The data was without form and null, > > and darkness was upon the face of the console; and the Spirit of IBM > > was moving over the face of the market. And DEC said, "Let there be > > regi
RE: :Oracle && Apache::DBI
Ian-- I very occasionally get these errors while using DBI and DBD::Oracle under mod_perl. I find that it generally happens when a random, perfectly good SQL statement causes the Oracle process dump the connection and write the reason to alert.log. Try doing the following: from your oracle home, run: > find . -name 'alert*' -print Go to that directory, read the alert files, and look through any corresponding trace files. The trace files contain the sql that actually cause the trace dump. I find that I can usually rewrite the sql statement in such a way that it no longer dumps core. Again, this happens _very_ rarely. Hope this helps, Ed -Original Message- From: Ian Kallen [mailto:[EMAIL PROTECTED]] Sent: Monday, May 22, 2000 9:37 PM To: [EMAIL PROTECTED] Subject: DBD::Oracle && Apache::DBI I've done everything I can think of to shore up any DB connection flakiness but I'm still plagued by errors such as these: DBD::Oracle::db selectcol_arrayref failed: ORA-12571: TNS:packet writer failure ...this is only a problem under mod_perl, outside of the mod_perl/Apache::DBI environment everything seems fine. Once the db connection is in this state, it's useless until the server gets a restart. My connect strings look good and agree, I put Stas' ping method in the DBD::Oracle::db package, set a low timeout, called Oracle (they don't want to hear about it). Everything is the latest versions of mod_perl/Apache/DBI/DBD::Oracle connecting to an Oracle 8.1.5 db on Solaris. Is Apache::DBI not up to it? (it looks simple enough) Maybe there's a better persistent connection method I should be looking at? -- Salon Internet http://www.salon.com/ Manager, Software and Systems "Livin' La Vida Unix!" Ian Kallen <[EMAIL PROTECTED]> / AIM: iankallen / Fax: (415) 354-3326
pod and EmbPerl
Does anyone know whether it is possible to pod-ify an EmbPerl document? When embedding pod directives in my EmbPerl pages and then running pod2html on them, the pod2html interpreter returns a blank page. thanks, Ed
RE: pool of DB connections ?
Oleg-- I don't know that this will help, but you could try using DBI::Proxy. The setup would theoretically be as follows: each apache process uses DBI::Proxy as the client; each creates a network connection to DBI::ProxyServer, which creates a few persistent connections to the db server using the connect_cached method. I have not tried this myself, but reference is made to it at http://www.crystaltech.com/perldocs/lib/site/DBD/Proxy.html For more help, you might try the DBI mailing list at http://www.symbolstone.org/technology/perl/DBI/ good luck -Ed > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of Oleg Bartunov > Sent: Monday, November 29, 1999 12:00 PM > To: Leslie Mikesell > Cc: [EMAIL PROTECTED] > Subject: Re: pool of DB connections ? > > > On Mon, 29 Nov 1999, Leslie Mikesell wrote: > > > Date: Mon, 29 Nov 1999 09:59:38 -0600 (CST) > > From: Leslie Mikesell <[EMAIL PROTECTED]> > > To: Oleg Bartunov <[EMAIL PROTECTED]> > > Cc: [EMAIL PROTECTED] > > Subject: Re: pool of DB connections ? > > > > According to Oleg Bartunov: > > > > > I'm using mod_perl, DBI, ApacheDBI and was quite happy > > > with persistent connections httpd<->postgres until I used > > > just one database. Currently I have 20 apache servers which > > > handle 20 connections to database. If I want to work with > > > another database I have to create another 20 connections > > > with DB. Postgres is not multithreading > > > DB, so I will have 40 postgres backends. This is too much. > > > Any experience ? > > > > Try the common trick of using a lightweight non-mod_perl apache > > as a front end, proxying the program requests to a mod_perl > > backend on another port. If your programs live under directory > > boundaries you can use ProxyPass directives. If they don't you > > can use RewriteRules with the [p] flag to selectively proxy > > (or [L] to not proxy). This will probably allow you to cut > > the mod_perl httpd's at least in half. If you still have a > > problem you could run two back end httpds on different ports > > with the front end proxying the requests that need each database > > to separate backends. Or you can throw hardware at the problem > > and move the database to a separate machine with enough memory > > to handle the connections. > > I didn't write all details but of course I already have 2 servers setup. > > Regards, > > Oleg > > > > > Les Mikesell > >[EMAIL PROTECTED] > > > > _ > Oleg Bartunov, sci.researcher, hostmaster of AstroNet, > Sternberg Astronomical Institute, Moscow University (Russia) > Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ > phone: +007(095)939-16-83, +007(095)939-23-83 >
load/regression test builders, monitoring tools for mod_perl apps
Does anyone know of any good open source test builders for regression/performance-testing a mod_perl app? This is the essence of what I would want such a suite to do: RECORD: -set up a proxy server to forward HTTP requests to a mod_perl'd server. -capture all GET/POST requests from the client and log them to a file, along with the server's output. The server's output would be the 'master' copy. PLAYBACK (REGRESSION): -play back the GET/POST requests and capture the output. Compare the output against the master copy. Raise an error in the log file if the two differ. PLAYBACK (LOAD): -play back the GET/POST requests according to some load scheme to see how well the application holds up under load. If this doesn't exist, I think it would be easy enough to write using LWP; I just don't want to duplicate anyone's efforts. I'd also be interested to know if anyone knows of any good webserver monitoring programs that could automatically kill spinning httpds, short of a CRON job. FYI-- I have encountered mystery spinning httpd's as well, but I have always been able to pin it down on bad/risk code or thrashing. At any rate, I still need to be able to kill spinning httpds should it come to that. cheers, Ed
weirdest bug i have seen in a long time: EmbPerl 1.2b10
First, I'd like to mention that I think EmbPerl is the greatest thing since sliced bread, and I'd like to thank Gerald for his time and effort. But here's an exceptionally strange bug that will cause the embperl/apache process to seg fault: ---BEGIN SCRIPT [- $hello %n; -] END SCRIPT- In fact, anything of the form: .+ %n.* will seg fault, but I have found no other combination (.* %t.*, etc.) that will seg fault. It's not a critical bug-- I don't know of any well-formed Perl statement that looks like that (I found it as a bug in my program)-- but it's a curious one. If anyone has an explanation, I'd be very interested in hearing the answer. My setup: Apache/1.3.9 (Unix) DAV/0.9.11 mod_perl/1.21 mod_ssl/2.4.2 OpenSSL/0.9.4 HTML::Embperl 1.2b10 Perl 5.00404 Linux kernel 2.0.36 (Red Hat 5.2) cheers, Ed
RE: sybase / mod_perl / linux question? [OFFTOPIC]
Interesting. I tested an identical setup of Apache/modperl/Embperl/Oracle on NT and Linux, and I experienced a huge slowdown on NT. When I looked into it, I found that the more database-intensive the page, the slower the relative performance of the NT platform. I took that to mean that it was actually Oracle on NT that was the root of the problem, but wasn't sure. I guess now I know. -Ed >Doesn't seem to have hindered it in any way for me. Linus has > discussed raw > partitions several times - the upshot being that they tend to be no faster > than using e2fs because e2fs is very fast anyway, and you'd simply have to > implement a lot of the e2fs functions inside of your database anyway. > > Anyhow - I don't know about all the internal workings of these things - > just that it works and is damn quick for me. > > BTW: Someone benchmarked Oracle on Linux vs Oracle on NT (using TPC code) > and found the Linux version to be about 4-7 (depending on the test) times > faster. So I guess that's a finger in the eye to raw partitions (I think > the NT version can use raw partitions - correct me if I'm wrong). > > -- > > > Details: FastNet Software Ltd - XML, Perl, Databases. > Tagline: High Performance Web Solutions > Web Sites: http://come.to/fastnet http://sergeant.org > Available for Consultancy, Contracts and Training. >
