Re: highscalability.com report
On 04/12/2012 10:00 AM, Clinton Gormley wrote: I think the bigger factor in the speed improvement is probably to do with switching from MySQL to Redis https://groups.google.com/group/redis-db/browse_thread/thread/77841c595d29f983?pli=1 They didn't really "switch" from mysql to redis. They are using Redis as a structured caching layer in front of mysql. Their authoritative data is still stored in mysql, but they have several "views" of that data that they export regularly to redis. The site pulls these cached views from redis and yes that's very fast. It's kind of similar to a lot of other really big systems (like facebook). Authoritative data in an SQL database and then their read data in denormalized structures in a NoSQL database. -- Michael Peters Plus Three, LP
Re: Question about caching
On 03/13/2012 03:51 PM, Dan Axtell wrote: The apps are mostly dynamic forms based on HTML::Template. H::T has some caching options but they seem more aimed at CGI users. I could convert to Template::Toolkit, which as I understand converts to compiled Perl code (which would presumably mean that mod_perl would leave the template objects in memory). Why do you think the HTML::Template caching options are geared towards CGI users? It's actually the opposite. All of the caching modes work just fine under mod_perl. You just need to pick whether you want normal cache (per-process memory), shared_cache (shared memory), double_cache (mix of normal cache and shared_cache) or file_cache (no extra memory uses the filesystem so slower). In fact, if you using normal CGI you can only use shared_cache or file_cache. -- Michael Peters Plus Three, LP
Re: Using Apache v1
On 02/28/2012 02:28 PM, Chris Olive wrote: Last I checked, Apache was still providing patches and releases for the v1 base as well. This is not the case. V1 has been "End of Lifed" and won't receive any new releases, not even security related ones. http://httpd.apache.org/docs/1.3/ There are a lot of people still using it, at least that is also my perception. This is probably true. We are stilling using V1 with mod_perl as an app server behind a V2 proxy. But we definitely have plans to change this because of the security implications of running something that will never receive any further security fixes. -- Michael Peters Plus Three, LP
Re: preloading modules and apache::dbi
On 01/19/2012 08:48 AM, Josh Narins wrote: The idea of Apache::DBI is that you get to pool connections. If you call Apache::DBI->new and there is a spare connection, you get it, if not, one is created for you. This isn't quite accurate. Apache::DBI doesn't do connection pooling, it manages persistent connections. It's still one connection per-process (per connection attributes) and connections aren't shared between child processes (that's where things get messy). Apache::DBI makes sure you don't create more connections than needed and makes sure that the persistent connections are still connected before handing them back. You almost certainly don't want one $db object being shared as a member of a class, unless your entire program also happens to represent one DB transaction. This is actually a pretty common and ok pattern. Remember that execution through a request is still linear (not threaded or handled by multiple processes) so if you want your whole request to be a single transaction you can, or if you want it to auto-commit you can, or if you want it to be a mix of transactions and auto-committed statements, you can do that too. When you use Apache::DBI and then call DBI->connect or DBI->connect_cached you'll get the same database connection every time (per process of course). -- Michael Peters Plus Three, LP
Re: Running multiple copies of same site in ModPerl
On 11/29/2011 10:29 PM, Nishikant Kapoor wrote: I have been running a ModPerl site fine with following configuration. The problem started when I made a complete copy of the original site and tried to run the two in the same environment. For some reason, calling up the ORIG site (http://127.0.0.1/ORIG/) now calls the COPY site at http://127.0.0.1/COPY/. None of the changes made in ORIG::MODS::base.pm are getting picked up even after restarting the apache server. However, changes to COPY::MODS::base.pm are showing up fine. The problem is that mod_perl uses the same Perl interpreter for everything (by default) and that interpreter is persistent. I'm guessing that your "package" declaration in ORIG::MODS::base.pm and COPY::MODS::base.pm are the same. Perl won't load 2 packages of the same name. The usual solutions are: 1) rename your modules so that they don't have the same package names. 2) run multiple separate apaches on the same physical server (not as hard as it seems to most people and pretty easy to do if you're running a proxy in front. And you really should be running a proxy in front). 3) Try to have multiple Perl interpreters using a different Apache MPM. I haven't seen anyone give a definitive guide to how to do this (although I could have missed something) and results seem to be mixed. I prefer #2 and use it constantly. It also makes it really easy to have separate dev environments each using their own code. -- Michael Peters Plus Three, LP
Re: Authentication logic [was: Changing browser URL based on condition]
On 07/14/2011 12:57 PM, Vincent Veyron wrote: This is what I first did, using Apache::Session. But I noticed the call to tie was very slow (response time around 70ms with it, 15ms without it), so I changed for Storable because filesystem reads were much faster. I don't personally like Apache::Session because of the tie thing, but that's more an interface preference than anything else. Also, I did not find how to store a hash in the database without tie. I read it's possible to use Data::Dumper to write the data in a field and read it as Perl code. Would that be a way to do it? The same way you're doing it now with Storable and a file. But instead of reading a file you read a database field. -- Michael Peters Plus Three, LP
Re: Best approach to store Application Configuration
On 07/11/2011 05:41 PM, Jerry Pereira wrote: please correct me if I am wrong, I should be using tool like YAML/Config::General for application configuration storage and reteieval, and load them on startup using startup.pl <http://startup.pl> script? Yes. That would mean i will have to store the name of configuration file some where (probabaly in mod_perl configuration block in httpd.conf). Again, if you think about just using a mod_perl specific way of doing this you'll leave all of your non-mod_perl stuff out in the cold and any project of significant size is going to have some non-mod_perl processes involved somewhere. I prefer to use environment variables if you need to specify the location of a config file. These are available no matter where you're running (in mod_perl you'll want to use a PerlPassEnv directive so the mod_perl side sees it). -- Michael Peters Plus Three, LP
Re: Best approach to store Application Configuration
On 07/11/2011 05:16 PM, James B. Muir wrote: This page describes pretty well how to set up custom configuration directives; perhaps helpful? http://perl.apache.org/docs/2.0/user/config/custom.html I would almost always avoid this kind of configuration and go with an external configuration file. Every project of any decent size will have some scripts or processes that don't run under mod_perl and thus can't use this apache-only configuration. As for configuration in Perl if I were starting a new project, I'd probably go with something like Config::Any and then pick a backend format. But in practice it probably doesn't matter a whole lot which config module you use as long as it's not tied to Apache. But if you like the apache-style format you can use Config::ApacheFormat which works well. -- Michael Peters Plus Three, LP
Re: Changing browser URL based on condition
On 07/11/2011 03:14 PM, Jerry Pereira wrote: Any suggestions to handle this scenario will be great. As others have noted, there isn't a way to do this. If it's a requirement of your application then the only way to handle it is to do redirection. And as others have pointed out it's a good idea to do a redirect after a POST anyway since it prevents other problems. -- Michael Peters Plus Three, LP
Re: mod_perl EC2 AMI's or other platform providers?
On 07/10/2011 06:28 AM, Tosh Cooey wrote: Looks there like they have a Perl stack available, which is super for the world but not so for me since the stack requires you use PSGI which is a great approach but since I don't require portability I never went that route, oh woe is me... PSGI isn't just about portability, it's also about middleware. With a common interface between components we get out of the rut where each new cool plugin was re-implemented by every existing framework. Now most things can just be written as PSGI middleware and anything can use it no matter what your framework (or lack of one). For instance, these are some handy ones to have: Plack::Middleware::InteractiveDebugger Plack::Middleware::REPL Plack::Middleware::Firebug::Lite Plack::Middleware::JSONP Plack::Middleware::RefererCheck Plack::Middleware::Static::Minifier Plack::Middleware::HTMLMinify Plack::Middleware::Mirror Plack::Middleware::iPhone And there's a ton more. Yes lots of them can be done with other existing Apache modules (but not all of them) and lots of them might already exist in your framework of choice. But pulling things out into middleware seems to be the future of such work and will thus get new features and be better maintained than lots of other alternatives. -- Michael Peters Plus Three, LP
Re: How do you use mod_perl for your web application?
On 06/16/2011 04:18 PM, Fred Moyer wrote: Maybe I'm not completely grokking how people are starting new projects using Plack, but it seems like the way to go is to not use Plack itself to write the code, but to use one of the many web frameworks (Mason2, Catalyst, Mojolicious) and then use Plack to specify what webserver is used. Plack is just middleware. Yes, but lots of people are using Plack to refer to the Plack family of stuff. Mainly the PSGI spec, the middleware and possibly one of the new Plack/PSGI oriented servers (like Starman). I see the role of mod_perl2 going forward as not something that applications are written on, but something that webserver middleware interfaces with. Yeah, that's what I see too. I'd like to see the performance of Starman vs mod_perl for normal applications (that don't need to do anything fancy with Apache). If it's anywhere close to mod_perl than I suspect lots of people would use it instead since it's much easier to setup and also much easier to package with your app since it's just a CPAN module. Would be nice to through FastCGI into that benchmark too. -- Michael Peters Plus Three, LP
Re: How do you use mod_perl for your web application?
On 06/16/2011 12:01 AM, Fred Moyer wrote: I'll start. I have a couple of Apache::Dispatch based applications I wrote. I also work on an Apache::ASP large codebase, and a couple of different Catalyst based systems. All are running on mod_perl 2.0.4 in production (the ops haven't upgraded to 2.0.5 yet). We use CGI::Application on mod_perl 1.31 (I know, I know) with Apache 2 as a reverse proxy. If I were to migrate, I would probably try out something like Mojolicious on Plack on mod_perl2. Performance of mod_perl2 has never been an issue to date, but I have Perlbal doing connection handling as a reverse proxy. We're looking at migrating to PSGI. We actually plan to do a full evaluation of backends (mod_perl2 vs Starman) and proxies (varnish, lighttpd, nginx, apache2, perlbal) with SSL thrown in the mix too (some proxies don't do SSL so we'll look at doing proxy + pound) -- Michael Peters Plus Three, LP
Re: CGI and character encoding
On 02/24/2011 04:31 PM, André Warnier wrote: I wonder if someone here can give me a clue as to where to look... The CGI.pm documentation talks about the -utf8 import flag which is probably what you're looking for. But it does caution not to use it for anything that needs to do file uploads. -- Michael Peters Plus Three, LP
Re: [ANNOUNCE] mod_perl 2.0.5
On 02/11/2011 12:02 PM, Randolf Richardson wrote: I use reload all the time it worked OK for most modules but some modules (at least the ones that use "use base 'modulename';" ) will have problems. That's very interesting because where I'm seeing the crashing usually involves code that uses "use base" somewhere (in one particular case the base is for some basic database stuff that utilizes "DBIx::Class"-auto-generated code). I highly doubt that it's tied to "use base" at all, but instead is caused by modules that are highly dynamic (do things like create methods at runtime like DBIx::Class would do). But because this magic is buried in the base class you don't see it until you use that base class as your parent (via "use base"). Perl doesn't have a real way to unload a module or to completely force a reload. Apache::Reload tries it's best but there are lots of modules it can't handle. It's not it's fault really, it's a feature that's missing in Perl. These days, I never use Apache::Reload. I just restart my dev server ( yes, I believe each dev should have their own dev Apache server). -- Michael Peters Plus Three, LP
Re: On memory consumption - request for comments
On 02/11/2011 11:01 AM, Hendrik Schumacher wrote: I didnt have time yet to read Torsten's post (will do later) but I will take a stab at this question. You are missing the difference between address space and used memory. Sample extract from /proc/*/smaps: Size:884 kB Rss: 536 kB Pss: 17 kB Shared_Clean:536 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 536 kB Swap: 0 kB In this case the address space has a size of 884 kb. Only 536 kb are used though (Rss/Referenced). All of this space is shared. As you can see the difference between 884 kb and 536 kb is not swapped out but just not allocated. Interesting. I didn't know that. But I think the questions that Torsten was posing about what would happen if an admin turned off swap while things were running doesn't apply then, right? This memory isn't in swap, it's just not in RAM. So turning off swap won't cause the shared/unshared sizes to blow up, right? -- Michael Peters Plus Three, LP
Re: On memory consumption - request for comments
On 02/11/2011 09:26 AM, Torsten Förtsch wrote: What does that mean? The total size of a process comprises its complete address space. Normally, by far not everything of this space is present in RAM. I'm not sure I'm understanding you correctly here, but are you saying that most of the time not all of a process's memory space is in hardware RAM? If you are saying that, then I would disagree, at least in the context of httpd/mod_perl. Most of the time (if not all the time) the goal should be to have enough RAM that your httpd processes completely fit into memory. Or am I missing something? -- Michael Peters Plus Three, LP
Re: experiencing Out of memory errors
On 01/27/2011 07:41 PM, Michael Ludwig wrote: Michael Peters schrieb am 27.01.2011 um 19:14 (-0500): But, even after all that I have applications where we consistently run 3-4G just for mod_perl/Apache. But surely not in one process as the OP said he'd like to do? No you're right, but I'm guessing he might be running a threaded MPM, so single process, multiple threads. -- Michael Peters Plus Three, LP
Re: experiencing Out of memory errors
On 01/27/2011 07:05 PM, Dave Hodgkinson wrote: Can I just say: WTF? 2G in an Apache? Surely there's a better way of architecting this? One thing to remember is that he's running Windows which doesn't have Copy-On-Write memory, so depending on what he's doing it might not take up as much memory if it were being run on Linux (or other OS with COW). Another thing that maybe the OP should look at (if he hasn't already) is to run a proxy in front of the main mod_perl application. Even if the proxy is on the same machine it will help because you can reduce the number of memory-heavy mod_perl processes/threads and handle the same number of connections. But, even after all that I have applications where we consistently run 3-4G just for mod_perl/Apache. -- Michael Peters Plus Three, LP
Re: Considering using Perl Sections
On 10/01/2010 04:47 PM, Chris Bennett wrote: At this point, should I break the AuthCookie sections out into confs for each virtual host using it, learn to use Perl Sections, or something different? Any suggestions are welcome. We actually like using templated httpd.conf files (we use HTML::Template, but you can easily use Template Toolkit, Text Template, etc). This lets us have really simple configs and then keep the stuff that's different for each client in a separate config file (or even the database) which is then pulled in and applied to the templated httpd.conf to generate the final conf. Works quite well for us. -- Michael Peters Plus Three, LP
Re: Using a handler other than 'handler'
On 09/27/2010 05:17 AM, Cédric Bertolini wrote: PerlAuthenHandler Authen::special doesn't work. It worked in mod_perl 1, but mod_perl 2 complains that it can't find Authen/special.pm <http://special.pm> in @INC. That's not what he suggested. He said: PerlAuthenHandler Authen->special Try that and see if it works -- Michael Peters Plus Three, LP
Re: PerlModule lifetime / threads
On 05/20/2010 08:54 AM, Some Guy wrote: The cleanup handler is a brilliant idea. That removes the need for a polling thread. If I attach the cleanup to the request's pool and run in a threaded MPM, this led to contention for the globals that I'm updating in C. Is this the case with PerlModules, or can I get away without locking them? Why use a threaded MPM? Is there something else you're running that relies on it? We generally recommend doing with the prefork mpm because you're running in share-nothing mode which performs better (and uses less memory on decent operating systems). -- Michael Peters Plus Three, LP
Re: Sinister variable caching problem with rand()
On 05/15/2010 04:47 PM, André Warnier wrote: A tip : to get a really unique identifier Another tip: to get a really unique identifier use mod_unique_id or Data::UUID or the UUID() function in mysql. -- Michael Peters Plus Three, LP
Re: best practie for open and close connection database
On 04/21/2010 12:33 PM, Felipe de Jesús Molina Bravo wrote: In my application, I opened my database from authentication handler and close it from child exit handler. It is correct (is a best practice)? or is a bad design? Typically it's best practice to not close DB handles. That's why Apache::DBI is such a popular module since it handles all of connection caching for you. With some databases (I'm looking at you Oracle) it's expensive to create a new database connection so it's more important. With others (like MySQL) it's really cheap so there's not a big penalty for not caching. There are some cases where caching connections is actually a bad thing: If you have too many application servers where each has a connection to your database and thus overwhelms your database with too many connections even though the actual load isn't overloading. But most people in this situation will use a proxy like DBD::Gofer to manage the DB connections. -- Michael Peters Plus Three, LP
Re: Usefulness of $r->notes and $r->pnotes (or $c)
On 03/23/2010 05:28 PM, Michael Ludwig wrote: What could be done at the connection level? Anything that might involve keep alive connections: where the same connection serves multiple requests. Probably not that useful for HTTP, but might be for other protocols. -- Michael Peters Plus Three, LP
Re: protecting internal redirects
On 03/18/2010 06:05 PM, Michael A. Capone wrote: This would be the most secure way. Saying it's the *most* secure way is a little stretch. It's *another* secure way. Also, keeping a large Perl/CGI process alive just to serve a static file is a waste. In fact, if you can think of a mod_rewrite rule to automatically look for the cached file first and send that before even getting to the CGI script would be your best bet for performance. -- Michael Peters
Re: protecting internal redirects
On 03/18/2010 04:59 PM, E R wrote: My question is: can the url which points to the cached results be protected so that it cannot be directly accessed by external clients? You should be able to do something like this for that block (so you might have to put that URL inside of a separate block) assuming the IP of your machine is 1.2.3.4 Order Deny,Allow Deny from all Allow from 1.2.3.4 -- Michael Peters Plus Three, LP
Re: CPAN Upload: P/PH/PHRED/Apache2-Dispatch-0.13.tar.gz
On 02/10/2010 08:45 AM, Mark Stosberg wrote: Is anyone here using this as an initial dispatcher to pass things on to CGI::Application::Dispatch? I've used both in different projects, but never together in the same project. -- Michael Peters Plus Three, LP
Re: Apache 1.3 end of life
Well, that was embarrassing :) Please ignore. -- Michael Peters Plus Three, LP
Re: Apache 1.3 end of life
This is the end for Apache 1.X, so we'll need to find some time to
upgrade to the Apache 2.X series for our Arcos backend server (the proxy
already runs Apache 2). This means upgrading mod_perl and might involve
some code changes (although hopefully not too much).
Don't need to do this super soon, but we should have it on our radar for
7.70 or so.
On 02/03/2010 10:13 AM, Adam Prime wrote:
FYI
The Apache Software Foundation and the Apache HTTP Server Project are
pleased to announce the release of version 1.3.42 of the Apache HTTP
Server ("Apache"). This release is intended as the final release of
version 1.3 of the Apache HTTP Server, which has reached end of life
status.
There will be no more full releases of Apache HTTP Server 1.3. However,
critical security updates may be made available from the following website:
http://www.apache.org/dist/httpd/patches/
see:
http://www.apache.org/dist/httpd/Announcement1.3.html
--
Michael Peters
Plus Three, LP
Re: mod_perl2 + fork + DBI = Chaos
On 01/25/2010 02:12 PM, Tosh Cooey wrote: Just to make things stranger, after I make a fork and things don't happen the way I expect them, subsequent calls to other programs running under mod_perl are generating Segmentation Faults. This could be due to the fact that the server is running in Europe and the SQL server in North America. I would really love to hear that this can cause segmentation faults because otherwise I have no idea where they are coming from This wouldn't cause segmentation faults. Network delays and timeouts yes, but not segmentation faults. But the overriding rule is that you can't use database handles opened before a fork in the child processes after the fork. You need to close those handles and reopen them. -- Michael Peters Plus Three, LP
Re: modperl / apache13 memory footprints on 32 adn 64 bit servers
On 12/20/2009 09:40 AM, Joe Niederberger wrote: 32bit: SIZE=45M RES=32M 64bit: SIZE=95M RES=45M Do these look reasonable? Should I worry? Good question. While I haven't made the jump to 64bit for my web machines there are lots of memory related things that double in size when you do so. I wouldn't expect it to be a full 2X jump, so something else might be amiss. Besides the 32v64bit change is all the other software the same? -- Michael Peters Plus Three, LP
Re: mod_perl works BUT rtld errors with some modules.
On 12/17/2009 04:25 PM, greg.geo...@orica.com wrote: BTW apache is starting as root and I have confirmed that startup.pl is being run as root by apache/mod_perl I am going to look in more detail at what DynaLoader.pm does in case this gives any clues. Not sure if this helps or not, but on some systems LD_LIBRARY_PATH can't be set by the same process that uses it. So you need it set by the process that starts the process that uses. So for instance, you need to make sure it's set in your shell environment and that it gets exported to the commands you run (sometimes "su" can remove ENV vars). And then you'll need a "PassEnv LD_LIBRARY_PATH" and a "PerlPassEnv LD_LIBRARY_PATH". But you can't do a "SetEnv LD_LIBRARY_PATH" in your httpd.conf. Not sure if that will help or not though since it's been a while since I've touched AIX. -- Michael Peters Plus Three, LP
Re: AJAX pseudo-push
On 11/12/2009 03:33 PM, Nicolas George wrote: If I can not do it with Apache and mod_perl, I will write my own HTTP server. But that is a shame, because it has to serve static content too, and that, Apache does definitely better than me. I wouldn't write your own. There are other event based, asynchronous web servers out there in Perl (HTTP::Server::Multiplex, Net::Server::Coro, Tatsumaki) so better to pick one of those than write your own. This technique in general falls under the umbrella term of "Comet", so I'd search CPAN for that. When I did it I quickly saw Stardust. -- Michael Peters Plus Three, LP
Re: Apache2::Request->param fails with POST
On 10/21/2009 06:12 PM, nmittal wrote: Hi I am using Apache2.2 and mod_perl. I have a login page that POSTs to an authentication script. Try not mixing GET and POST parameters. Put that sid as a hidden input in the form instead of a GET parameter on the query string? Technically you're not supposed to mix the 2, but most systems allow it anyways. But maybe it's tickling a bug here. -- Michael Peters Plus Three, LP
Re: Custom INC per-directory
On 10/20/2009 05:24 PM, Perrin Harkins wrote: Performance will suffer, but it will work for most code. Right. Your Perl interpreter will be persistent, but none of your code will be cached and thus every module you use will be reloaded on every request. It also means you can't do pretty standard things like preloading modules on start up. And if you're going to have your own dispatch class, doing your own module loading, why is having separate namespaces a big deal? > It will fail > on the same kind of things that Apache::Reload fails on. This usually affects modules that dynamically add methods to classes (like most ORMs, Moose, etc). -- Michael Peters Plus Three, LP
Re: Custom INC per-directory
On 10/20/2009 01:06 PM, Devin Teske wrote: I've been trying for what feels like months now to find some way to have two directories specify the same module name to PerlInitHandler but to use two different modules. This isn't a limitation of mod_perl but a limitation of Perl. You can't have 2 different modules with the same name. This comes up from time to time on this list and the best advice usually comes down to this: Run multiple apaches (on the same machine if you want). Then put a proxy in front to redirect to the right one based on the host or path or whatever. This is just slightly more complicated to set up, but is the most flexible and doesn't require as many resources as it initially appears. -- Michael Peters Plus Three, LP
Re: Configuring virtual hosts on the fly
On 10/13/2009 12:17 PM, Scott Gifford wrote: I have had mixed experiences in the past with automatically restarting Apache after a configuration change. It is very easy to end up with something unexpected in the configuration, which causes the configuration to fail, which causes apache to stop. And with a cluster, that would happen on all hosts. If you run apache with a -t option it will check that the configuration is sane. I'm assuming that your configuration is auto-generated by this monitor somehow, then you can create a new config file and test it with httpd -t -f /path/to/new/file Then if it's ok, you can move it to the real location and do the restart. When I have done this in the past, I have done it with generating configuration files, so of course one misplaced newline or angle-bracket will kill the server. I generally prefer generated config files, especially if you have multiple sites with basically the same config. I also use templated config files so that the template I'm working on looks a lot like the finished product. It's easier to prevent typos this way, IMO. -- Michael Peters Plus Three, LP
Re: Configuring virtual hosts on the fly
Looking at this from a different perspective, have you tried writing a monitoring program that looks for updates to the database and then would restart the appropriate apache servers on the various machines. It would do them one at a time (taking them out of rotation from your load balancer). It wouldn't be immediate response, could definitely be done in under a minute. -- Michael Peters Plus Three, LP
Re: Mason or Registry?
On 09/27/2009 02:48 AM, 叶孤城 wrote: One of our customers run a small site (about 3000 IP each day), written with Perl CGI originally. Now to expand some features and improve the performance, they want to upgrade the site. So do you think which is better that it will be re-written with pure modperl (for example, using Mason) If you are going to do a rewrite I would suggest a more modern framework. Most people that use Mason these days use it as a templating engine and that can even be done inside of a framework like Catalyst or CGI::Application. , or keep CGI and move them to modperl Registry? This is probably the biggest bang for the buck. I wouldn't recommend it as a way to start a new application, but for migrating an existing application its a very good choice. -- Michael Peters Plus Three, LP
Re: problem with CSS in a dynamic document
On 09/25/2009 08:17 AM, Chuck Crisler wrote:
# output a document
print $query->header();
print $query->start_html(-title=>"Howdy",
-style=>{-src=>'./dynamic.css'});
print $query->h1('Form Data');
Also, not to confuse you too much, but most people don't use CGI.pm's
HTML generation features any more. Most of us have moved on to using
HTML templates which makes it even easier to separate things out. And
it's more like editing a normal HTML file which is what most designers
are familiar with.
You should check out Template Toolkit
(http://search.cpan.org/perldoc?Template) or HTML::Template
(http://search.cpan.org/perldoc?HTML::Template).
--
Michael Peters
Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 12:57 PM, Randal L. Schwartz wrote: The problem with that is public web browsers. You *cannot* ensure the expiration of an auth cookie, so you'll have to have some sort of server-side data to say "this user most recently authenticated at this time, so I still trust him". Why does this have to be server side? Why can't it be part of the cookie's (tamper proof) data itself? -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/18/2009 12:16 PM, Ihnen, David wrote: Its security through obscurity admittedly - security in that you can't see my code, methodology, or shared secret configuration. No it's not really through obscurity. Even if someone found out your method of serialization your data is still safe. It's only if they find out your secret key that you'll have problems. But that's the same for SSL, PGP and any other crypto. -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 11:19 AM, Randal L. Schwartz wrote: Yes. Welcome to phase 2. Eventually, you'll enter phase 3. I used to be a phase 3 convert and then some really smart people convinced me otherwise :) -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 11:15 AM, James Smith wrote: But cookies are in general not big enough to store the information that a user would store on a website! I'm not talking about eliminating a permanent data store for your users. I'm talking about replacing the session specific things. How much session specific data do you really need to store? If it's bigger than 4K per-user than yes you can't use a single cookie. But like I said before, the situations that you really need more than that for *session specific* data are pretty rare. and security is not just on your server (but also on the clients machine) so if the browser can read it - anyone that can compromise the browser can also read it - if it is on the server that is harder! It's almost as if people aren't reading my other emails :) If you need to prevent tampering, use a hash. If you need to completely secure the data, encrypt it. -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/18/2009 11:13 AM, Tina Mueller wrote: How does the user invalidate that "session"? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? When you generate the hash for the cookie, you can also include the timestamp and the IP address of the client. If the cookie leaks it can't be used (unless the person who steals it is also on the same NAT'd network and uses it quickly). But you'll have that same problem anyway. Is one select per request that bad? if the website is completely dynamic you will probably have other requests as well? One extra select on every request can add up. In most web architectures the DB is a scarce shared resource. If you care about the number of selects you should IMHO better safe those with the help of caching. Caching of sessions could help, but if you don't need to go down that road, why do it in the first place? -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 10:33 AM, Randal L. Schwartz wrote: Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. I disagree. Using cookies for session data has a lot of advantages: + No need to have a permanent data store (DBD::Deep is single server only and why waste the resources to go across the network to your DB or cache if you don't have to). Also no need to clean up this data store periodically. Having a single source for this data also kills scalability. + If it's commonly used data, putting it into a cookie will make it available to the client side Javascript. Why waste server resources to do what the client's machine can do. In fact, I find it's more flexible to have this information in the cookie since my front end folks can then use it to add functionality without having to trouble the back end folks. And securing a cookie is pretty easy. If the information is not sensitive then you just need to put a hash in it to make sure it's not tampered with. If it is sensitive, then encryption works for cookies too. I'm not saying there aren't uses for large server side sessions, but I think they are pretty few. -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 09:14 AM, Igor Chudov wrote: I realized that there is something I am missing. Sometimes I may need to put something into the session after I did $cgi->start_html. I can do it if the cookie is only a session ID, with session data stored in mysql. This might be a larger architectural problem them. IMO, you really shouldn't use CGI.pm for HTML generation. Most people use a template of some sort for that. And the next thing would be to reorganize your code so that you aren't tied to a specific order of doing things. For instance, I normally use CGI::Application which lets me do all kinds of things to the headers and content until I'm all ready to return at the end of my run mode. The order doesn't matter. But how can I change the cookie AFTER I called $cgi->start_html? Since cookies are part of the HTTP headers they need to go out before any HTML content. So instead of just printing the content as you go, why don't you collect your HTML into a variable and then print it all out at the end. -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/16/2009 01:02 PM, Igor Chudov wrote: I actually do cache generated pictures, I store them in a database table called 'bincache'. This way I do not have to compute and draw every image on the fly. If I have a picture in bincache, I serve it, and if I do not, I generate it and save it. That saves some CPU, but makes mysql work harder. Then don't put it in your database. A cache is not a permanent store and it's usage patterns will be different than a database. I'd either use a real cache like memcached or have your proxies cache them. In addition to that you can send the appropriate HTTP cache headers so that browsers themselves will never request that image again. Make the client machine do the caching. -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/16/2009 12:48 PM, Adam Prime wrote: I have very little static content. Even images are generated. My site generates images of math formulae such as (x-1)/(x+1) on the fly., I can understand generating them on the fly for flexibility reasons, but I'd cache them, and serve them statically after that, rather than regenerate the images on every single request. Definitely good advice. Especially if your images are generated the same each time and never change. For instance, I don't think the image generated by the formula "(x-1)/(x+1)" would ever change (unless you changed your application code and in that case you can clear you cache). -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/16/2009 12:13 PM, Brad Van Sickle wrote:
Can I get you to explain this a little more? I don't see how this could
be used for truly secure sites because I don't quite understand how
storing a hash in a plain text cookie would be secure.
If you need to store per-session data about a client that the client
shouldn't be able to see, then you just encrypt that data, base-64
encode it and then put it into a cookie.
If you don't care if the user sees that information you just want to
make sure that they don't change it then add an extra secure hash of
that information to the cookie itself and then verify it when you
receive it.
I like to use JSON for my cookie data because it's simple and fast, but
any serializer should work. Something like this:
use JSON qw(to_json from_json);
use Digest::MD5 qw(md5_hex);
use MIME::Base64::URLSafe qw(urlsafe_b64encode urlsafe_b64decode);
# to generate the cookie
my %data = ( foo => 1, bar => 2, baz => 'frob' );
$data{secure} = generate_data_hash(\%data);
my $cookie = urlsafe_b64encode(to_json(\%data));
print "Cookie: $cookie\n";
# to process/validate the cookie
my $new_data = from_json(urlsafe_b64decode($cookie));
my $new_hash = delete $new_data->{secure};
if( $new_hash eq generate_data_hash($new_data) ) {
print "Cookie is ok!\n";
} else {
print "Cookie has been tampered with! Ignore.\n";
}
# very simple hash generation function
sub generate_data_hash {
my $data = shift;
my $secret = 'some configured secret';
return md5_hex($secret . join('|', map { "$_ - $data->{$_}" } keys
%$data));
}
Doing encryption and encoding on small bits of data (like cookies) in
memory will almost always be faster than having to hit the database
(especially if it's on another machine). But the biggest reason is that
it takes the load off the DB and puts it on the web machines which are
much easier to scale linearly.
> I know a lot of true app servers (Websphere, etc..) store
this data in cached memory,
You could do the same with your session data, or even store it in a
shared resource like a BDB file. But unless it's available to all of
your web servers you're stuck with "sticky" sessions and that's a real
killer for performance/scalability.
--
Michael Peters
Plus Three, LP
Re: Ways to scale a mod_perl site
On 09/16/2009 11:49 AM, Igor Chudov wrote: 1) Use a load balancer like perlbal (I am already doing that) A load balancer is good but so are proxies. If you can separate your application server from the server that servers static content then you'll get a boost even if they are on the same machine. 2) Separate a MySQL database server from webservers. This is probably the first and easiest thing you should do. 3) Being enabled by item 2, add more webservers and balancers 4) Create a separate database for cookie data (Apache::Session objects) ??? -- not sure if good idea -- I've never seen the need to do that. In fact, I would suggest you drop sessions altogether if you can. If you need any per-session information then put it in a cookie. If you need this information to be tamper-proof then you can create a hash of the cookie's data that you store as part of the cookie. If you can reduce the # of times that each request needs to actually hit the database you'll have big wins. 5) Use a separate database handle for readonly database requests (SELECT), as opposed to INSERTS and UPDATEs. Use replication to access multiple slave servers for read only data, and only access the master for INSERT and UPDATE and DELETE. Reducing DB usage is more important than this. Also, before you go down that road you should look at adding a caching layer to your application (memcached is a popular choice). -- Michael Peters Plus Three, LP
Re: Fwd: Undefined symbol "Perl_pad_sv" building mp2
cr...@animalhead.com wrote: I have always included previous perl libraries in the @INC of new builds. And it has always worked, with the single exception of building mod_perl2 this time. All of the perl scripts and modules on my site work well under the new 5.10.1. Pure Perl modules should work between versions (they might have some new warnings, but they should work), but anything with XS (C code) isn't guaranteed to work across major versions. So when you've upgraded between say 5.8.2 and 5.8.7 you wouldn't need to re-install. But when upgrading from 5.8.X to 5.10.X you will need to re-install. Aside from the pain of re-downloading all kinds of modules as they prove to be needed over the next year or so, I know there are items in the 5.8.7 libraries from my Internet Hosting Provider, that are needed to run software from the IHP. If you need 5.8.7 libs that you can't re-compile against 5.10.1 then just stick with 5.8.7. I don't see a command to tell the mod_perl build process to use a particular perl. If I say $perl_other Makefile.PL in the modperl-2.x directory, does that do it? Yes, the perl you use when you run the Makefile.PL is the one that's used in the compilation of mod_perl. -- Michael Peters Plus Three, LP
Re: RFC: Apache2::CloseKeepAlive
Perrin Harkins wrote: It's not a run time option. The subject is a mod_perl2 script, that can only be used within the Apache2 server. I can't conceive why the DB name would ever change, and multiple programs won't use the module. Some people have many apache installations on one machine, or may install new ones after installing your module. This is the main point I guess I didn't clearly express. Don't assume that just because you have only 1 apache installed that everyone else does too. Also, if you deviate from what the standard install tools do (which is install things based on how CPAN is configured on that machine) then actually make it harder for people who need to install your module in a non-standard place. This is pretty common for dev environments. I (and most Perl people I would assume) already know how to tell EU::MM or Module::Build how to put things where I want them. If you do it differently it makes our lives harder. I guess what it comes down to is that putting modules into non-standard places is pretty common, so please don't implement your own way of doing that. -- Michael Peters Plus Three, LP
Re: RFC: Apache2::CloseKeepAlive
cr...@animalhead.com wrote: The installation process needs to know at least: 1. where the accompanying DB-building script should go Module::Build knows where to put bin/ or script/ files and it does so according to how Perl and CPAN are configured. I assume EU::MM does too. 2. where the DB that the app builds should go That's probably something your application should allow to be specified, either a command line option to the script or an option to the module, or both. How can such necessary things be determined, other than by asking the user? If it's a run time option (and can be changed or multiple programs can use this and thus want different values) don't ask for it at install time. I was just going to add a note before or after these two queries, that if they have a mod_perl2 library and want the module to go there, they should "do this". Putting extra docs in your module is ok, but I personally wouldn't repeat the docs that are already out there. As I just wrote to Perrin off-list, since only a tiny fraction of CPAN downloads are mod_perl2 modules, no one can configure CPAN for a mod_perl2 library. All perl libraries should go in the same place IMO. Whether it's Apache specific, or GTK or something else. I don't want to have to mess with @INC just because someone put something someplace non-standard. So now if Makefile.PL doesn't find 'mm' in /usr/local/lib, it prompts the user to enter the path to 'mm', which is then passed to Devel::CheckLib. The query is accompanied by a note showing the URL from which 'mm' can be downloaded. Personally, I'd just bomb out if you can't find the lib (and Devel::CheckLib is good for that). Give the user a message that you can't find the library and then exit. I doubt that someone who has a library that's not in the system libs will know where it is, and if they do they can just as easily set LD_LIBRARY_PATH themselves. On the other hand, my latest version 0.58 has only been tested by one tester, a week after uploading it. Maybe that's because of the new query in Makefile.PL! That's why prompts are evil. Automated tests won't work. So just bomb out and at least you'll get lots of UNKNOWN test results which is better than nothing :) -- Michael Peters Plus Three, LP
Re: RFC: Apache2::CloseKeepAlive
cr...@animalhead.com wrote: Including a note about how to do something in an installation dialog, for people who wouldn't otherwise know, is not controlling anything. Please don't put a dialog in your installation process! CPAN is supposed to be automatic after it's been configured. There are some rare occasions where it makes sense to ask some questions, but where you want the module installed is not one of them. CPAN takes care of that. -- Michael Peters Plus Three, LP
Re: At random moments, mod_perl starts returning empty pages
Fred Moyer wrote: The issue is that at random moments, usually once or twice a day, my webserver starts returning completely empty pages instead of actual content. The pages are mod_perl based. I've seen this happen sometimes when something segfaults. And segfaults are usually the result of having incompatible binaries/libs on your system. Is there anything in your error logs about this? -- michael peters I played golf... I did not get a hole in one, but I did hit a guy. That's way more satisfying. - Mitch Hedberg
Re: share use vars?
Brad Van Sickle wrote: I'm also wondering why this needs to be shared between processes... but if it is required, you can share variables between mod_perl processes by caching them on server startup, assuming that they are read only. Calling this variable "shared" is a little misleading. If you're using a system with copy-on-write memory, then yes the physical memory is shared by the processes, but that's invisible to the users and the variable is not logically shared. Also, there are lots of things that could happen to make that page in memory "dirty" which would cause the physical RAM to become unshared. You don't have to actually change the variable for this to happen. But the sentiment is right and this is a great way to fake the sharing of a lot of read only data. I didn't recommend this technique because there is a big caveat: file handles. I don't know the implementation details of Geo::IP but it does pull it's data from data files. If it slurps it all at once into memory and then just uses that data afterwards then you're ok. Or if it opens a new filehandle every time it wants data and then closes it you're also ok. But if it opens a filehandle and then keeps it around you can't share it. All sorts of bad and weird things can and will happen in very unpredictable ways when inherited filehandles are used by forked children. -- Michael Peters Plus Three, LP
Re: share use vars?
Fayland Lam wrote: I'm wondering in which way the $geo_ip is sharable through the threads. we are using "server/mpm/prefork" Neither. Prefork doesn't use threads it uses separate processes. And processes cannot share Perl variables like that. Both of the examples you showed would only have a single Geo::IP object per-process (although you probably don't want to use a global like that. A local (my) or package visible (our) variable is probably better. Why do you want to share it between your processes? -- Michael Peters Plus Three, LP
Re: Conf. multiple apache locations with colliding namespaces
Perrin Harkins wrote: You can also run separate mod_perl backends I usually like this approach since it gives you a lot of control and let's you manipulate (start, stop, etc) one instance without having to affect others. It also gives you more scaling options as you can in the future easily split off some clients to other boxes as things grow. Just put a simple proxy in front of all these mod_perl processes to make things perform better and reduce the number of mod_perl processes you need. But all of these processes does mean more RAM, so make sure you have lots. If you can a 64bit box will let you cram in more. Memory will almost definitely be your bottleneck since Perl sometimes trades memory for speed. -- Michael Peters Plus Three, LP
Re: ErrorDocument from a registry script
Perrin Harkins wrote: Really? It looks to me like he never got it working. He said: If I just set "status => 404" with CGI.pm / Apache::Registry and return nothing, it works the first time, and then after that I get a lot of these errors And further on in the conversation it seemed like those errors he was getting were unrelated to the 404/ErrorDocument part of it. But I'm wondering if his "it works" was just Apache sending it's default 404 document and not the one in the custom ErrorDocument. -- Michael Peters Plus Three, LP
Re: ErrorDocument from a registry script
Adam Prime wrote: I don't have a mod_perl 1.3 setup handy, but i was able to get it to do what you're wanting to do just fine in mod_perl 2 requesting the script results in me seeing the error document found at /404.html Maybe it's a mod_perl 1.3 limitation. I do get Apache's internal 404 document (not the browser's) just not my custom ErrorDocument. The HTTP status header is also the same between the requests. -- Michael Peters Plus Three, LP
Re: ErrorDocument from a registry script
Perrin Harkins wrote: You also need to return the correct code from the handler. This is not simple to do from Registry. Have a quick look at the list archives for things related to Registry and 304/404 handling. Apache::RedirectLogFix might be the ticket for you. I'd rather not have to go that route if possible, just seems a little messy for this one small problem (and it seems that A::RLF doesn't return anything but OK so it's not apparent that returning NOT_FOUND would really work there). Looking at this email thread (http://perl.markmail.org/message/yimclxzgspjtibbn?q=Apache::Registry+404+ErrorDocument#query:Apache%3A%3ARegistry 404 ErrorDocument+page:1+mid:xtfrl2hywqugoqlh+state:results) it seems that it worked (mostly for Mark Stosberg) so there's a glimmer of hope. -- Michael Peters Plus Three, LP
Re: ErrorDocument from a registry script
Michael A. Capone wrote: In the spirit of checking off the obvious things... are you sure that the ErrorDocument directive is working for other pages? in other words, can you open a browser and goto http://mydomain.com/this_file_does_not_exist.htm and see your ErrorDocument?\ Yes, that's how I know that it's not the browser silently ignoring my ErrorDocument (which IE is known to do). Has anyone else done this before or can someone confirm to me that it should work? Would help to know that I'm not chasing an impossible dream. -- Michael Peters Plus Three, LP
ErrorDocument from a registry script
I'm pretty sure this should just work and I'm doing something stupid, so any helpful insults that point me in the right direction would be appreciated :) I have a registry script that could return a NOT FOUND (404) error. I'm handling that like so: Apache->request->status(NOT_FOUND); The browser does get the 404 HTTP status header, but it does not trigger the ErrorDocument 404 that is set up, instead just showing the browser's built-in 404 message. PerlSendHeaders is Off and I'm not printing any output. So what am I missing? -- Michael Peters Plus Three, LP
Re: GoogleBot
jmcaric...@greta-besancon.com wrote: 66.249.65.228 - - [07/Apr/2009:15:01:41 +0200] "HEAD /sitemap.xml.gz HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.65.228 - - [07/Apr/2009:15:01:44 +0200] "GET /sitemap.xml.gz HTTP/1.1" 302 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Why 2 requests The first is a HEAD request (http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods) and why my server return status 302 and not 200 ? Seems you have some sort of Auth handler in front that does a redirect (which is what a 302 is). If you want to find out why you should try hitting that resource with your browser pretending to be the Googlebot. If you're using Firefox you should look at the User Agent Switcher plugin. -- Michael Peters Plus Three, LP
Re: google's web music player
Jeff Pang wrote: How google achieve this? Thanks. Completely off topic. Next time please label your subject with an OT so we're know. But I suspect they are using flash (which most music players use) and flash has it's own cookie system which is outside of the browsers. -- Michael Peters Plus Three, LP
[ANNOUNCE] Krang 3.05 released
The Krang development team is proud to announce the release of version 3.05. Krang is an open source CMS, particularly well suited for high-volume magazine-style web sites. And because it has an extremely flexible addon system, developers can customize it to create a CMS that is specific to your industry. Originally developed in 2003 by Primedia, Krang is used by many well-known web sites including New York Magazine, Motor Trend, and Soap Opera Digest. Screenshots available at https://sourceforge.net/project/screenshots.php?group_id=103045 This release mostly includes small features and bug fixes. For full details please see the change log located at https://sourceforge.net/project/shownotes.php?release_id=668591&group_id=103045 -- Michael Peters Plus Three, LP
Re: get backtrace emailed when code dies running under ModPerl::RegistryPrefork?
Adam Prime wrote: This was news to me, so i went to the docs. According to them, you have to explicitly enable this behavior. see: http://perl.apache.org/docs/2.0/api/Apache2/Log.html#C_Apache2__Const__LOG_TOCLIENT_ Is that still accurate? I don't know about mod_perl2, but I didn't have to do anything to have it enabled under mod_perl1. -- Michael Peters Plus Three, LP
Re: get backtrace emailed when code dies running under ModPerl::RegistryPrefork?
Perrin Harkins wrote:
I'm not sure if that's accurate or not, but setting a $SIG{__DIE__}
handler should work fine.
You don't even need to use a __DIE__ handler. mod_perl will place any runtime
errors into $r->notes('error-notes'). I don't like my errors going to the
browser (only really works if your content is HTML anyway) but for the email I
usually use an Apache cleanup handler. That way it gets sent after the client
request has returned, so they don't have to wait on it.
I use something like this:
# in httpd.conf where you need it
PerlCleanupHandler MyProject::ApacheCleanup
# in MyProject/ApacheCleanup.pm
package MyProject::ApacheCleanup;
use Apache::Constants qw(:common);
sub handler ($$) {
my ($class, $r) = @_;
return DECLINED unless $r->is_main;
my $status = $r->last->status;
return DECLINED unless $status == SERVER_ERROR;
my $error = $r->notes('error-notes') || $ENV{ERROR_NOTES};
return DECLINED unless $error;
# create your email message and send it. I also like to put in a
# Data::Dumper dump of %ENV, the URL, timestamp and other project
# specific config settings
return DECLINED;
}
--
Michael Peters
Plus Three, LP
Re: Deployment strategies...
Carl Johnstone wrote: 1) We want to run different versions of the same app for different sites - this means I have a namespace problem for MyApp v1 vs MyApp v2 etc. Just my opinion, but it's sooo much easier to manage multiple sites if they are all using the same version of the application. What are your reasons for wanting to do that? -- Michael Peters Plus Three, LP
Re: lexical scoping under mod perl solution
Erik Aronesty wrote: (A solution would be to force all "my" varialbes at the file scope to be "our" and then put the cgi in an anonymous package:) A file/package-scoped "my" var is not the same thing as an "our" var. One example is that a "my" var can't be seen outside of the package/scope it's in. A package level "our" var can be seen. -- Michael Peters Plus Three, LP
Re: WELCOME to modperl@perl.apache.org
Erik Aronesty wrote: mod_perl should be *extremely* verbose in pointing this change out to developers, as it is *extremely* difficult to diagnose. Maybe ModPerl::Registry should do this, but not mod_perl itself. If someone is trying to get an old CGI script to work under mod_perl then they'd use Registry. But there's no reason to make the rest of us suffer for their old scripts. Alternatively, it should be fixed. But I'm not sure how, since most solutions involve creating anonymous package scopes which require the recompiling and ruining of mod_perl's speed benefits. No, changing your code's structure or scope is not something that should be done automatically. Maybe there should just be a big fat warning on the top of the CGI module and at the top of mod_perl, "Don't use file-scoped lexicals!" Or something like that. The problem with that is that they are useful for having things like Singleton's that don't change between invocations of the script (like a Conf object). -- Michael Peters Plus Three, LP
Re: Using mod_perl in a non httpd environment
titetluc titetluc wrote: My questions: is there a way to run mod_perl/my_module on an HTTP server/reverse proxy other than apache ? If so, what kind of HTTP server/reverse proxy ? mod_perl is an Apache module written to the Apache API. Like every other Apache module, it will only work under Apache. The only exception would be if someone wrote a compatibility layer between some other HTTP server and Apache, but that would almost certainly not save you any memory. -- Michael Peters Plus Three, LP
Re: Class method or Object method?
Mark Hedges wrote:
package Cart;
my $instance;
sub new { return $instance ||= bless {}, $self }
sub is_in_cart { ... }
Under mod_perl, doesn't that mean that if user A is the
first to hit an interpreter and the cart instance is created
for them, then if user B's request is the next to be served
by the mod_perl child, that user B will get user A's cart?
Yes. mod_perl (or any persistent Perl framework) doesn't clear out globals or package variables when
the request is done. And in this case, $instance is a package variable.
If that's what you want, then it's a valid way to implement it (usually called
a Singleton pattern).
--
Michael Peters
Plus Three, LP
Re: Class method or Object method?
kropotkin wrote: In general is it better to use a class method or object method? It depends on what you're using it for and how. Does the method act on any instance specific state data? Or does it act on class specific data? A more concrete example would be a Dog class. Is bark() a method on Dog or $fido? Well, every dog, no matter it's name is going to bark the same. Is eat() a method on Dog or $fido? Well, if eat() changes $fido's next_poop_interval property, then obviously it needs to be an instance method. And of course, if your system is only every going to have 1 Dog then you don't need to ever have instances, right? -- Michael Peters Plus Three, LP
Re: article about mod_perlite
macke...@animalhead.com wrote: IMO the profile below does not indicate that the sky is falling. A survey of "how long have you been using computers?" might show a similar distribution. Just because it might have the same distribution doesn't mean it's relevant. I bet a question like "How long have you been having sex?" would get a similar curve :) A more relevant question for comparison would be "How long have you been using PHP?" from a PHP community survey. While I have my doubts about mod_perlite, I think the quest to bring more new blood into the Perl community is necessary and important one. Without it we won't get the next generation of people I'll need to hire :) -- Michael Peters Plus Three, LP
Re: Initializing Sleepycat::DbXml (Berkeley, Oracle) objects in startup.pl
cr...@animalhead.com wrote: I have a vague recollection of reading about the circumstances in which filehandles can be inherited, but can't remember where. I've been bitten by this a few times. Filehandles (and thus sockets) are inherited across forks. If your system isn't very busy you won't notice a problem. But when things heat up you will start to get very strange problems. And I do mean strange. I was doing something with Inline::Java (which communicates over a socket to a JVM) under mod_perl and the communication was getting all garbled. Multiple processes were sending and reading off the socket which confused the other processes since they were all getting partial reads on their data and partial reads from the data for other processes. But the problems only happened during the busiest 2 weeks of the year and were fine the rest of the time. But those weren't very fun weeks :) -- Michael Peters Plus Three, LP
Re: which reverse proxy for modperl?
Perrin Harkins wrote: I haven't seen a good comparison that hits all the popular proxy servers (perlbal, pound, nginx, lighttpd, apache, squid... I think I'm forgetting some) but I've wanted one before. If you could include varnish, I'd be really happy :) -- Michael Peters Plus Three, LP
Re: Best filesystem type for mod_cache in reverse proxy?
Raymond Wan wrote: I had looked at the effect compression has on web pages a while ago. Though not relevant to modperl, there is obviously a cost to compression and since most HTML pages are small, sometimes it is hard to justify. Not to discredit the work you did researching this, but a lot of people are studying the same thing and coming to different conclusions: http://developer.yahoo.com/performance/rules.html Yes, backend performance matters, but more and more we realize that the front end tweaks we can make give a better performance for users. Take google as an example. The overhead of compressing their content and decompressing it on the browser takes less time than sending the same content uncompressed over the network. I'd say the same is true for most other applications too. As for dialup, if I remember from those dark modem days :-) Even non dialup customers can benefit. Many "broadband" connections aren't very fast, especially in rural places (I'm thinking large portions of the US). But all this talk is really useless in the abstract. Take a tool like YSlow for a spin and see how your sites perform with and without compression. Especially looking at the waterfall display. -- Michael Peters Plus Three, LP
Re: Best filesystem type for mod_cache in reverse proxy?
Adam Prime wrote: That does look like a big deal, if i were in your situation, I'd try running with only mod_deflate, then only mod_cache, and see what happens. There are benefits to running the reverse proxy alone (without mod_cache), so that'd be the first scenario i'd try. Or split them up. If you have any static assets that can benefit from mod_deflate (Javascript, CSS, etc) then put mod_deflate on the proxies and mod_perl, mod_cache on the backend. -- Michael Peters Plus Three, LP
Re: Best filesystem type for mod_cache in reverse proxy?
Perrin Harkins wrote: On Mon, Nov 24, 2008 at 3:16 PM, Michael Peters <[EMAIL PROTECTED]> wrote: Well except for getting 15K disks you probably won't be able to get much more improvement from just the hardware. You don't think so? RAID and SSD can both improve your write throughput pretty significantly. He's already using RAID0, which should be the best performance of RAID since it doesn't have to use any parity blocks/disks right? And from what I've seen about SSD (can't find a link now) filesystems haven't caught up to it to make a real difference with one over the other. They do have much lower powser usage though (which is why they find their way into laptops). -- Michael Peters Plus Three, LP
Re: Best filesystem type for mod_cache in reverse proxy?
Michael Peters wrote: According to these benchmarks (http://fsbench.netnation.com/new_hardware/2.6.0-test9/scsi/bonnie.html) ReiserFS handles deletes much better than ext2 (10,015/sec vs 729/sec) But these benchmarks (http://www.debian-administration.org/articles/388) say the following: For quick operations on large file tree, choose Ext3 or XFS. Benchmarks from other authors have supported the use of ReiserFS for operations on large number of small files. However, the present results on a tree comprising thousands of files of various size (10KB to 5MB) suggest than Ext3 or XFS may be more appropriate for real-world file server operations But they both say don't use ext2 :) -- Michael Peters Plus Three, LP
Re: Best filesystem type for mod_cache in reverse proxy?
Neil Gunton wrote: Perrin Harkins wrote: On Mon, Nov 24, 2008 at 2:42 PM, Neil Gunton <[EMAIL PROTECTED]> wrote: The section on "Maintaining the Disk Cache" says you should use htcacheclean, which is what I've been doing, and it doesn't seem to be up to the job. I can't speak to your filesystem question but you might consider getting better disks. Either a RAID system or a SSD would help your write speed and both are pretty cheap these days. I'm using 4x10k SCSI drives in RAID0 configuration currently, on an Adaptec zero channel SmartRaid V controller. Filesystem is ext2. Well except for getting 15K disks you probably won't be able to get much more improvement from just the hardware. According to these benchmarks (http://fsbench.netnation.com/new_hardware/2.6.0-test9/scsi/bonnie.html) ReiserFS handles deletes much better than ext2 (10,015/sec vs 729/sec) -- Michael Peters Plus Three, LP
Re: Apache 1.3 -> 2.x portability
Phil Carmody wrote: I'm currently running only Apache 1.3 (mod_perl 1.22) and am wondering what kinds of issues there might be if I were to want to migrate my server to Apache 2.x and mod_perl 2. Are there any things that I can do (or avoid) to make such a migration as close to trivial as possible. http://perl.apache.org/docs/2.0/user/porting/compat.html The biggest problem that I've seen is that not all of the Apache:: modules on CPAN have been ported to Apache2 or been setup to work for both. Most of the big ones have so it depends on what you're using. And it's not that hard to port one of them if you need to. -- Michael Peters Plus Three, LP
Re: mod_perl survey results
Hmm, this is making me want to run benchmarks! Maybe a solid set of benchmarks would be a fun OSCON presentation next year. ++ I've loved your other comparison talks in the past and this would be a nice one. Make sure to include the new Mojo (kind of like Mongrel but in Perl). -- Michael Peters Plus Three, LP
Re: rewrite urls to modperl handler
[EMAIL PROTECTED] wrote:
I have mod rewrite setup to redirect request that don't an existing
file or directory to the perl handler.
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^/(.*) /ex_perl_h [L]
but this doesn't work.
Why do you mean by "this doesn't work". There are lots of ways for it to work incorrectly. Should we
guess? Ok, my guess is that's it's going to /ex_perl_h but not /ex_perl_h/docName. Try changing your
RewriteRule to this:
RewriteRule ^/(.*) /ex_perl_h/$1 [L]
--
Michael Peters
Plus Three, LP
Re: $r->status no correct
Geoffrey Young wrote: I'd check your logs but the common log format uses %>s. try $r->last->status() Aha! That's it, thanks a lot Geoff! -- Michael Peters Plus Three, LP
Re: $r->status no correct
Ryan Gies wrote: Below is an Apache log snippet which traces the handler phases for two requests: A) /scratch/ does NOT have a directory index B) /scratch/foo/ DOES have a directory index (index.html) So here is mine (the text for die() is "A horrible, terrible death!") A) /foo/index.pl B) /foo/ A [Wed Oct 29 17:46:06 2008] [warn] phase=PerlPostReadRequestHandler, status=200, uri=/foo/index.pl, filename= [Wed Oct 29 17:46:06 2008] [warn] phase=PerlTransHandler, status=200, uri=/foo/index.pl, filename= [Wed Oct 29 17:46:06 2008] [warn] phase=PerlHeaderParserHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:06 2008] [warn] phase=PerlAccessHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:06 2008] [warn] phase=PerlTypeHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:06 2008] [warn] phase=PerlFixupHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:06 2008] [error] A horrible, terrible death! [Wed Oct 29 17:46:06 2008] [warn] phase=PerlLogHandler, status=500, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl CLEAN URI: /foo/index.pl STATUS: 500 main: 1 initial: 1 [Wed Oct 29 17:46:06 2008] [warn] phase=PerlCleanupHandler, status=500, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl B [Wed Oct 29 17:46:35 2008] [warn] phase=PerlPostReadRequestHandler, status=200, uri=/foo/, filename= [Wed Oct 29 17:46:35 2008] [warn] phase=PerlTransHandler, status=200, uri=/foo/, filename= [Wed Oct 29 17:46:35 2008] [warn] phase=PerlHeaderParserHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo [Wed Oct 29 17:46:35 2008] [warn] phase=PerlAccessHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo [Wed Oct 29 17:46:35 2008] [warn] phase=PerlTypeHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo [Wed Oct 29 17:46:35 2008] [warn] phase=PerlFixupHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo [Wed Oct 29 17:46:35 2008] [warn] phase=PerlPostReadRequestHandler, status=200, uri=/foo/index.pl, filename= [Wed Oct 29 17:46:35 2008] [warn] phase=PerlTransHandler, status=200, uri=/foo/index.pl, filename= [Wed Oct 29 17:46:35 2008] [warn] phase=PerlHeaderParserHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:35 2008] [warn] phase=PerlAccessHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:35 2008] [warn] phase=PerlTypeHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:35 2008] [warn] phase=PerlFixupHandler, status=200, uri=/foo/index.pl, filename=/tmp/test_client_pub/foo/index.pl [Wed Oct 29 17:46:35 2008] [error] A horrible, terrible death! [Wed Oct 29 17:46:35 2008] [warn] phase=PerlLogHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo/ [Wed Oct 29 17:46:35 2008] [warn] phase=PerlLogHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo/ CLEAN URI: /foo/ STATUS: 200 main: 1 initial: 1 [Wed Oct 29 17:46:35 2008] [warn] phase=PerlCleanupHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo/ [Wed Oct 29 17:46:35 2008] [warn] phase=PerlCleanupHandler, status=200, uri=/foo/, filename=/tmp/test_client_pub/foo/ So "/foo/" goes through some extra steps (lines 2-7) but after that it looks the same as "/foo/index.pl" until you hit the die(). Then the status for "/foo/index.pl" goes to 500 while the status for "/foo" stays at 200. -- Michael Peters Plus Three, LP
Re: $r->status no correct
Adam Prime wrote:
You should package that up and put it on CPAN.
Attached is a version that works on mod_perl 1.
--
Michael Peters
Plus Three, LP
package Apache::Trace;
use strict;
use Apache::Log();
use Apache::Constants ':common';
our @Phases = qw(
PerlPostReadRequestHandler
PerlTransHandler
PerlMapToStorageHandler
PerlHeaderParserHandler
PerlAccessHandler
PerlAuthenHandler
PerlAuthzHandler
PerlTypeHandler
PerlFixupHandler
PerlResponseHandler
PerlLogHandler
PerlCleanupHandler
);
sub handler($$) {
my ($pkg_name, $r) = @_;
_add_http_phase_tracers($r);
DECLINED;
}
sub _add_http_phase_tracers {
my $r = shift;
for (@Phases) {
my $phase_name = $_;
$r->push_handlers($_, sub($) {
my $r = shift;
my $s = $r->server;
$s->warn(sprintf('phase=%s, status=%s, uri=%s, filename=%s',
$phase_name,
$r->status,
$r->uri,
$r->filename,
));
return DECLINED;
});
}
}
1;
__END__
=pod:summary Trace mod_perl server events
=pod:synopsis
In your Apache configuration file:
PerlPostReadRequestHandler Apache2::Trace->http_phases
Will produce output (C) for each http request, e.g.:
... [warn] phase=PerlPostReadRequestHandler, status=200, uri=/, filename=
... [warn] phase=PerlTransHandler, status=200, uri=/, filename=
... [warn] phase=PerlMapToStorageHandler, status=200, uri=/,
filename=/var/www/vhosts/example.com/htdocs/
... [warn] phase=PerlHeaderParserHandler, status=200, uri=/,
filename=/var/www/vhosts/example.com/htdocs/
... [warn] phase=PerlAccessHandler, status=200, uri=/,
filename=/var/www/vhosts/example.com/htdocs/
... [warn] phase=PerlTypeHandler, status=200, uri=/,
filename=/var/www/vhosts/example.com/htdocs/
... [warn] phase=PerlFixupHandler, status=200, uri=/,
filename=/var/www/vhosts/example.com/htdocs/
... [warn] phase=PerlLogHandler, status=304, uri=/index.html,
filename=/var/www/vhosts/example.com/htdocs/index.html
... [warn] phase=PerlCleanupHandler, status=304, uri=/index.html,
filename=/var/www/vhosts/example.com/htdocs/index.html
=cut
Re: $r->status no correct
Ryan Gies wrote: Below is an Apache log snippet which traces the handler phases for two requests: This is probably a really dumb question, but how do you get a log like that? I tried setting "LogLevel debug" but no dice. -- Michael Peters Plus Three, LP
Re: $r->status no correct
Michael Peters wrote: No, I just get URI: /foo/ STATUS: 200 Nothing else. The browser get's a 500 and the access log reports 500, but the actual request gets a 200. I even tried it from a log handler just to see if I get the same thing. Does it matter that I'm using Apache::Registry to process index.pl? -- Michael Peters Plus Three, LP
Re: $r->status no correct
Adam Prime wrote: Just to be clear, when you do this: GET /foo/ do you get this in the error log? URI: /foo/index.pl STATUS: 500 (the sub request from Directory Index) URI: /foo/ STATUS: 200 (the original request) No, I just get URI: /foo/ STATUS: 200 Nothing else. The browser get's a 500 and the access log reports 500, but the actual request gets a 200. -- Michael Peters Plus Three, LP
Re: $r->status no correct
Ryan Gies wrote: On Wed, 29 Oct 2008 15:29:18 -0400 Michael wrote: So what's going on here. Shouldn't the 2 requests (/foo and /foo/index.pl) be treated exactly the same? /foo is much different than /foo/index.pl because /foo is handled by mod_dir (http://httpd.apache.org/docs/2.0/mod/mod_dir.html). Ok then let's talk about "/foo/" and "/foo/index.pl" so that we're not talking about the redirect. return Apache2::Const::OK unless $r->is_initial_req; I'm now have the following for debugging: warn "URI: " . $r->uri . " STATUS: " . $r->status . " main: " . $r->is_main . " initial: " . $r->is_initial_req . "\n"; When run for /foo/index I get: URI: /foo/index.pl STATUS: 500 main: 1 initial: 1 When run for /foo/ I get: URI: /foo/ STATUS: 200 main: 1 initial: 1 So they both show up as initial requests and the main request. Plus if it was subrequests that were the problem I would have expected to see more than debug statement in the log since the cleanup handler was getting more than once, right? -- Michael Peters Plus Three, LP
Re: $r->status no correct
Adam Prime wrote: Michael Peters wrote: But when I make the request to just /foo (instead of /foo/index.pl) I only get this in my error log: URI: /foo/ STATUS: 200 you've said to /foo, but your error_log is saying /foo/. What's going on there? mod_dir redirecting a bare directory request? Sorry about that. Seems we have a redirect for /foo to /foo/, but the problem still remains if I do a request for /foo/ direct (no involving the rewrite). Is this happening with just a mod_perl enabled server, and no proxies or anything? Yes, I'm making requests directly against the mod_perl server not the proxy. I even tried to see if maybe $r->is_main changed for those 2 requests, but they are exactly the same. Does the client see the 200, or the 500? The client always sees the 500 which is what the access log reports as well. It's just that in the CleanupHandler it comes back as 200 for /foo/, but 500 for /foo/index.pl. -- Michael Peters Plus Three, LP
$r->status no correct
I've been banging my head against on wall on this one and could use a little
help.
Setting the Stage:
I've got an Apache::CleanupHandler that is running for my requests. I've got a file at
DocumentRoot/foo/index.pl and DirectoryIndex index.pl. Now, to test out error handling in my cleanup
handler I've put an explicit "die" at the top of foo/index.pl.
Problem:
When I make a request to /foo/index.pl every behaves correctly. I get this in
my access log:
127.0.0.1 - - [29/Oct/2008:15:19:33 -0400] "GET /foo/ HTTP/1.1" 500 644 "-" "Mozilla/5.0 (X11; U;
Linux i686; en-US; rv:1.9.0.2) Gecko/2008092318 Fedora/3.0.2-1.fc9 Firefox/3.0.2"
And this in my error log (where I'm printing $r->uri and $r->status from inside
the cleanup handler):
URI: /foo/index.pl STATUS: 500
But when I make the request to just /foo (instead of /foo/index.pl) I only get
this in my error log:
URI: /foo/ STATUS: 200
The stranger part is that the access log still reports it correctly as an error:
GET /foo/ HTTP/1.1" 500
And the error still appears in the error log, it's just that $r->status (and
$r->notes('error-notes') don't contain the right information).
So what's going on here. Shouldn't the 2 requests (/foo and /foo/index.pl) be
treated exactly the same?
--
Michael Peters
Plus Three, LP
Re: Reducing memory usage using fewer cgi programs
Carl Johnstone wrote: Something else I will do for my low-usage but "massive" scripts (those that have large memory structures and take several seconds to execute) is to place these in a non-mod_perl directory so I can be assured their memory usage goes away at the end of the response. <<<<< There's no reason not to run these under mod_perl too - any memory allocated by perl will be re-used. This is only true if those structures were created during run time and go out of scope at run time. If they are generated at compile time or attached to global variables or package level variables, they will not be re-used by Perl. -- Michael Peters Plus Three, LP
Re: reading apache configuration via mod_perl
Adam Prime wrote: Is it theoretically possible to actually generate something like that? > Any ideas? I'm definitely ignorant enought about Apache's internals to be able to answer that question. And the following doesn't really answer your question, but this is how we handle having a dynamic config while still being able to see what it looks like. We use a template. We have a wrapper script around Apache that queries the database and config files and then starts Apache pointing to the generated conf file. This gives us the benefit of a data driven config but we still get a real config file out of it that we can inspect. -- Michael Peters Plus Three, LP
Re: Reducing memory usage using fewer cgi programs
Thomas Hilbig wrote: I have about a dozen small cgi programs under mod_perl2 that all pretty well look like this.. use CGI; use DBI; use perlchartdir ; # graphing To think about how this works under mod_perl, pretend that all of your scripts are put together into 1 larger script and all those "use" statements are repeated. Does having multiple "use CGI" statements make your script use more memory? No. CGI.pm is only loaded once. Under mod_perl, will the memory footprint of the libraries be shared across all of the programs that use them The specifics depend on what OS you're running, what version of Apache/mod_perl you're running, but it's basically like this: Each Apache child has it's own Perl interpreter and what is loaded in that interpreter is persistant across requests. So different scripts can use the same CGI.pm or DBI that you've loaded for another script. Combining them all into the same program won't make any noticeable difference in memory one way or the other. I also use a PerlRequire startup.pl to preload most modules (CGI, DBI), but I thought that was only for quicker startup times and not for sharing memory. Is that correct? Preloading helps with speed (you don't get the the initial loading hit for a module the first time it's used in a specific process) but it can also help with memory on certain OSs. For instance, Linux has Copy-On-Write memory so that if you preload modules it saves on actual physical RAM used (even though the separate processes think they have their own separate memory spaces). But remember that each Apache child get's it's own perl interpreter. So if you have a high MaxClients you will run out of memory. It's basically Perl Memory * MaxClients for how much RAM could be used if your system got busy. This is one of the reasons that most people put a vanilla Apache (or something else like squid, lighttpd, varnish, etc) in front as a Proxy. When you do that, even if you're running both the proxy and the mod_perl server on the same physical machine you need a lot less RAM then if you just ran a mod_perl server trying to do static and dynamic requests. HTH -- Michael Peters Plus Three, LP
Re: Any success with storing photos in a database? (prevents double-submits)
Mark Stosberg wrote: So how might an implementation look? I would either make the uuid the primary key (might affect performance since it's not an integer, but a string) or a unique key for the same table. Then you don't have anything else to keep track of (no extra tables, etc). Every so often, the table could cleaned up via cron, (since we probably don't care about seeing the same UUID weeks apart, just seconds or minutes apart). UUID's should never collide. There is still room for a small race condition in between checking to see if we used the UUID and inserting it, but I think that may be acceptable. If you're really worried about someone attacking you in this way then insert the record with the uuid first and then let them upload. If you don't find the uuid they are trying to upload to, then they changed it so just disallow the upload. -- Michael Peters Plus Three, LP
Re: Any success with storing photos in a database? (prevents double-submits)
Mark Stosberg wrote: At one point in the past I did a variation of this where we put the next ID of a related database sequence in the form, and this would become the new primary key when inserted, and it would of course not allow the same primary key to be used twice. That worked, but I realized was open to abuse if a user tweaked the number to be larger than the sequence. Then, eventually a legitimate user would eventually be assigned that value by the sequence, and it would fail. Are their specific modules that you recommend to help with this? You can try a UUID (look at Data::UUID). It's not sequential, but someone could still tweak the form to create a value that could potentially conflict with another value in the future. But I'd say it's much less likely than a sequential id. -- Michael Peters Plus Three, LP
